22 Dec 2016

Beta Testing using Feature Flags

LaunchDarkly Feature Toggle Beta

It’s best practice for products to have some sort of beta – a way to collect customer feedback and test performance before releasing to everyone. In an era of continuous delivery, we are delivering new features and experiences more frequently and with less time to gather thorough customer and performance feedback. With this increased cadence, product teams are having to make betas shorter, forego them altogether, or slow down their release cadence to gather adequate customer feedback.

Challenges of traditional betas:

  • Coordinating Opt-Ins: It sometimes takes weeks or months to gather customer opt-ins to test new betas. You also have to organize the distribution of beta keys (ex. for early access to games) and reminder emails.
  • Organizing Focus Groups: Getting feedback from focus groups is often time consuming and expensive, creating a long feedback loop that lengthens the release process.
  • Opt-Out: If customers opt-in to a beta and don’t like the experience, then they will want a simple mechanism to switch back to the production version.
  • Granular Betas: It is very difficult to do targeted betas based on user attributes or to perform incremental percentage rollouts of new beta features.

Feature toggles

To overcome these challenges, smart product teams are beginning to run betas with feature flags/toggles. These are mechanisms for granularly controlling software releases, allowing you to control the timing and visibility of a beta release.

Currently, many betas are tied to code releases and are managed by a config file or database.  This approach requires engineering time or custom mechanisms to opt-in users.

With feature toggles, you can empower product, marketing, sales, and even customers (themselves) to opt-in new to a new beta experience.

Feature Toggle Beta Test LaunchDarkly

In this simple example, you can use a toggle to control the visibility of a new beta feature. Ideally, this toggle would be part of a user interface that could be controlled by a non-technical team member. The code, itself, could be deployed off and then turned on via the toggle.

Beta Test Percentage Rollout with Feature Toggle LaunchDarkly

Moreover, you can also use the toggle to control the percentage of users who get the beta experience. For instance, you could release the new beta experience and have it rolled out to 0% of users. You could gradually increase the rollout percentage from 1% to 5% to 20% and more, collecting customer and performance feedback along the way.

Surfacing this beta control functionality in a user interface is critical for giving non-technical team members access to release controls.

Regional betas

For a recent example of a targeted rollout, we can look at how Pokémon GO released their product country by country: first to the United States and then abroad.

This is a great use case for feature toggles because you can create targeting rules to determine which users receive the feature first. For example, I could create a toggle that is governed by the rule: “If users live in San Francisco, then serve the new Nearby Pokémon feature”. This allows you to maintain different regional feature sets without having to deploy different versions of the application. It also allowed Pokémon GO to refine their algorithms and assess customer feedback before rolling out the new feature to a wider audience.

Benefits of beta testing with feature toggles

  • Empowered non-technical users: Allow the sales, marketing, product, design, and business teams to turn features on for specific users, collect feedback, and control the business logic. This also substantially cuts down on engineering time.
  • Production feedback for your beta tests: Test features in production with limited user segments to collect customer and performance feedback.
  • Incremental percentage rollouts: Gradually roll out features to incrementally test performance and mitigate risk. If the feature is bad, toggle it off.
  • Real-time opt-in / opt-out: Allow users to opt in and out of beta tests in real time, controlled via the feature toggle. Skylight provides a nice article on this.

Getting started with toggles

Conceptually, a feature toggle is relatively simple. You create a conditional in your code that controls the visibility of a code snippet. There are many open source libraries that will allow you to get started.  However, these libraries become cumbersome when you try to feature toggle at scale or restrict access to particular toggles. Depending on your needs, you could consider a feature toggle management platform to provide a system for access control and mitigating technical debt.

14 Sep 2016

Launched: Custom Targeting Rules

LaunchDarkly custom rules targeting with feature flags

A goal of LaunchDarkly is to provide our customers with the ability to target your users to meet the needs of your unique use cases.  We have always allowed you to target individual users or groups by any attribute that you define: name, e-mail, group, ID, country, age, etc.  However, we did not let you perform granular percentage rollouts for each attribute, nor did we allow you to construct conditional targeting rules. Many of you have asked for this!

Now, we are excited to launch custom targeting rules, allowing you to construct a conditional rule using a user attribute, operator, and value.

Use Cases

Using both boolean and multivariate feature flags, you can perform extremely powerful and granular user targeting:

  • Roll out a new feature to users who registered before a certain date
  • Perform a percentage rollout for all of your users who live in Canada and the United States
  • Release a feature to your beta testers whose e-mail address ends with gmail.com
  • Turn on a VIP checkout system for all users who spend more than $100 per month
  • Release a new feature to 10% of users who belong to your Gold Plan and who signed up before August 1st, 2016
  • Target all of your users above age 35, and show the one-click checkout to 30%, the two-click checkout to 20%, and the original checkout to 50%.

Custom Targeting Rule

For a more concrete example, let’s say we have a feature flag that controls our app’s checkout flow.  This multivariate flag has three variations: one click checkout, two click checkout, and original checkout.  Here, we are performing a percentage rollout that targets all users whose email ends with gmail.com or yahoo.com:

LaunchDarkly feature flags and toggles using custom conditional targeting rules

Constructing a Rule

Let’s now build a rule using LaunchDarkly.  Here is a rule that serves the ‘true’ variation to all users whose e-mail address ends with gmail.com:

LaunchDarkly Custom Targeting Rule

You can target users by using any attributes that you send to LaunchDarkly, like group, country, age, and device.  For each feature flag, you can create as many custom targeting rules as you would like.

Additionally, you can add multiple conditions to a rule.  Users must meet all the conditions in a rule to match the rule. If any of the conditions are not met, the user will not match the rule.

Here, we’ve created a custom rule with two conditions. This rule serves true to all users whose e-mail address ends with “gmail.com” and whose country is either “USA” or “Canada”.

LaunchDarkly feature flags / toggles custom rule with multiple conditions

If we want to do a percentage rollout instead, we can select “percentage rollout” from the dropdown and allocate users accordingly.  Here, we have are serving “true” to 25% of users whose ‘groups’ attribute is “beta_testers”.

LaunchDarkly feature flags / toggles percentage rollouts using attributes

There are many exciting ways to use custom rules, especially when you take full advantage of our multivariate feature flags.  If you have any questions or feedback, please send them to support@launchdarkly.com and we’ll be happy to help!