22 Dec

A New Way To Beta Test

LaunchDarkly Feature Toggle Beta

It’s best practice for products to have some sort of beta – a way to collect customer feedback and test performance before releasing to everyone. In an era of continuous delivery, we are delivering new features and experiences more frequently and with less time to gather thorough customer and performance feedback. With this increased cadence, product teams are having to make betas shorter, forego them altogether, or slow down their release cadence to gather adequate customer feedback.

Challenges of traditional betas:

  • Coordinating Opt-Ins: It sometimes takes weeks or months to gather customer opt-ins to test new betas. You also have to organize the distribution of beta keys (ex. for early access to games) and reminder emails.
  • Organizing Focus Groups: Getting feedback from focus groups is often time consuming and expensive, creating a long feedback loop that lengthens the release process.
  • Opt-Out: If customers opt-in to a beta and don’t like the experience, then they will want a simple mechanism to switch back to the production version.
  • Granular Betas: It is very difficult to do targeted betas based on user attributes or to perform incremental percentage rollouts of new beta features.

Feature toggles

To overcome these challenges, smart product teams are beginning to run betas with feature flags/toggles. These are mechanisms for granularly controlling software releases, allowing you to control the timing and visibility of a beta release.

Currently, many betas are tied to code releases and are managed by a config file or database.  This approach requires engineering time or custom mechanisms to opt-in users.

With feature toggles, you can empower product, marketing, sales, and even customers (themselves) to opt-in new to a new beta experience.

Feature Toggle Beta Test LaunchDarkly

In this simple example, you can use a toggle to control the visibility of a new beta feature. Ideally, this toggle would be part of a user interface that could be controlled by a non-technical team member. The code, itself, could be deployed off and then turned on via the toggle.

Beta Test Percentage Rollout with Feature Toggle LaunchDarkly

Moreover, you can also use the toggle to control the percentage of users who get the beta experience. For instance, you could release the new beta experience and have it rolled out to 0% of users. You could gradually increase the rollout percentage from 1% to 5% to 20% and more, collecting customer and performance feedback along the way.

Surfacing this beta control functionality in a user interface is critical for giving non-technical team members access to release controls.

Regional betas

For a recent example of a targeted rollout, we can look at how Pokémon GO released their product country by country: first to the United States and then abroad.

This is a great use case for feature toggles because you can create targeting rules to determine which users receive the feature first. For example, I could create a toggle that is governed by the rule: “If users live in San Francisco, then serve the new Nearby Pokémon feature”. This allows you to maintain different regional feature sets without having to deploy different versions of the application. It also allowed Pokémon GO to refine their algorithms and assess customer feedback before rolling out the new feature to a wider audience.

Benefits of beta testing with feature toggles

  • Empowered non-technical users: Allow the sales, marketing, product, design, and business teams to turn features on for specific users, collect feedback, and control the business logic. This also substantially cuts down on engineering time.
  • Production feedback for your beta tests: Test features in production with limited user segments to collect customer and performance feedback.
  • Incremental percentage rollouts: Gradually roll out features to incrementally test performance and mitigate risk. If the feature is bad, toggle it off.
  • Real-time opt-in / opt-out: Allow users to opt in and out of beta tests in real time, controlled via the feature toggle. Skylight provides a nice article on this.

Getting started with toggles

Conceptually, a feature toggle is relatively simple. You create a conditional in your code that controls the visibility of a code snippet. There are many open source libraries that will allow you to get started.  However, these libraries become cumbersome when you try to feature toggle at scale or restrict access to particular toggles. Depending on your needs, you could consider a feature toggle management platform to provide a system for access control and mitigating technical debt.

29 Nov

Toggle Talk with Damian Brady

I sat down with Damian Brady, Solution Architect at Octopus Deploy for a conversation about his experience with feature toggles.  He shared with me his tips for best practices, philosophies on when to flag and what he thinks the future of feature flagging will look like. 

  • How long have you been feature flagging?

I had to think about this one a bit – about 8 years ago but I probably didn’t know what it was called at the time.

“It’s definitely the case that people are doing this without knowing the name “feature flag” or even giving it a name. They’re just saying it’s a configuration switch or a toggle and but not giving it a more proper name, they’re not identifying it as a first-class citizen really.”   

  • What do you prefer to call it and why?

Now I call it feature flagging or occasionally feature toggles. I think toggles makes a bit more sense as analogy for non-technical people.  

  • When do you think feature flagging is most useful?

There’s a couple – but the one I think it’s most useful for is to use a feature flag when you have a feature that is nearly complete or complete from your point of view. Either way, you are ready to get verification from someone with real data.

“You can test as much as you want with your pretend fake data, or even a dump from production which is being obfuscated, but until it gets used in the wild you’re never really sure that the feature is doing exactly what it needs to do.”  

So hiding that behind a feature flag, and then clicking it on for somebody who is using the product for real in any way gives you that last little test that is ultimately the most useful.  At that point you still have the opportunity to back out. If something was corrupt or your expectations were wrong, it’s really useful for that last-minute check.  

At Octopus, we’ve started using feature flags for big features that a lot of people don’t want to see. So a while ago we introduced the idea of a multi-tenant deployments. And probably most of our users don’t need that feature because it adds a lot of complexity to the UI.  We have a configuration section where you can toggle an “on” and “off” switch, so if you don’t need that feature you can just leave it off.     

Are there any cases where feature flagging is not a good idea?

I think there are two extremes where feature flagging is not a good idea. On one hand, flagging really small changes can be more trouble than it’s worth. It’s introducing an extra level of complexity that maybe for a small change is not critical.  

On the other side, using feature toggles around the architectural changes in the core of your application – that’s kind of hard to test. Do you have a feature flag that when you turn it “on” it completely redirects the way the entire application will run? In that case you bite the bullet and decide that this is a big change and you’re just going to have to test it very thoroughly and not give yourself a way out.

That being said, there are some cases where you still need to give yourself a way out by using a flag. For example, you might deploy some new feature thinking it’s correct, but subsequently learn from a customer or user that it doesn’t really meet their needs. Rather than the user living with a bad feature, you might want to turn the flag off and go back to the drawing board.

If it’s an architectural change, you may only find out that there’s a bug when you use it in production. Test data may not surface the issue properly.

Ultimately, doing core architecture changes in a way where you can back out later can be an extra huge amount of work. It’s probably at that point you know you aren’t going to do it (revert back) anyway.   

  • Best use of a feature flag – a personal story?

When I first started using feature flags, around the 8 years ago timeframe, I was working on a web application that was internal and a big line of business.  And we had just added a new third party provider for providing SMS.  And with this new provider, it meant we had to write a lot of new code.  It was internet banking software so it was a one-time password we were sending out – and it was really really important that it work.

We tested everything rigorously but wanted more insurance.  So we put the new service behind a feature flag. We had a bunch of agents that ran this type of SMS. We enabled a flag for one of the agents and monitored it to make sure it was actually doing the right thing and not failing. And then we started trying other ones. It failed a couple of times because of differences in the sandbox environment between the third-party provider and the real one.

“We thought everything was okay, but when we put it live we turned it on slowly, and it didn’t do what we expected.”

So when that happened we turned it back off again…and went back to the drawing board.

So without the feature flag, we would have dropped every person using the service at that critical point. That client would have not been able to receive SMS’s until we were able to rollback.

  • What do you think is the number one mistake that’s made around feature flagging?

There is one that I keep seeing – when you wrap a new feature you believe to be finished in a flag, the biggest mistake with this is not testing that change with the flag “on” and then “off”. For instance, when you turn it “on” it snaps into new database tables or starts changing the way data is saved. But when you turn it “off” again, you’ve lost that data or data is corrupt. For this you need to test it “on” and “off”.  

“If you have more than one feature flag running at the same time, test the combinations of them being both ‘on’ and ‘off’.

If they’re likely to interact with each other you need to test “one on, one off,” “both on,” “both off” and all possible combinations like this.  

  • How do you think feature flags play into the DevOps movement? What about Continuous Delivery?

I think feature flags play in both continuous delivery and continuous deployment. I think they’re most useful to continuous deployment. You have all of your features pushed out to production as soon as they compile essentially – but they are behind a feature flag so you don’t break anything. That’s the way Facebook does it. They know that any new code they write might end up in production so it’s going to be safe behind a feature flag.  

“The design of the DevOps movement, the aim of it really, is to get real features and real value in the hands of users as quickly as possible.”

So if you have to wait until this “half done piece of work” is actually safe to deploy then that slows you down. So having it behind a feature flag so that it doesn’t get touched until you are ready to test it can be really powerful for increasing velocity and getting things out to production much faster.     

So even for marketing teams, it means they don’t have to tell the developers “hey we worked out the result of this a/b test and we want option b.” If the marketing department can just just flip that switch and say “no, option b is working better so just leave it there” without a new deployment or contacting the developers to remove the old stuff and redirect to the new stuff,  that increases that team effort of getting value to customers which is the whole purpose of DevOps.

  • Can you share any tips for better flagging?

If you’re feature flagging a big change, pair the feature flag with a branch by abstraction pattern. See the clip from my talk from NDC Sydney for more details.

There’s also the concept of transitional deployments – again refer to my video clip here for more. It’s useful for things like database schema changes where you have a midpoint for both the new and old applications that will work with the schema that’s currently there. So you can turn that feature off if you need to.

  • Are you seeing feature flagging evolving? If so how?  And how do you expect it to change in the future?

It’s been around for a long time…but I think it’s becoming much more visible – and partly it’s LaunchDarkly helping with that. I think more people will start using feature flags in their continuous delivery pipeline. And the more continuous delivery becomes mainstream, the more mainstream developers will need feature flags.  

“I think feature flagging is starting to be something that you have to add your deployment cycle because you know it needs to be fast and you know you feature needs to get to production as quickly as possible – and feature flags are the way to do that.”  

So as it becomes more mainstream I think there will be more tools, more frameworks, more awareness of it (feature flags) as it hits more and more companies. I think there will be things coming out like feature flag-aware testing tools – so testing tools that know that they need to test with this flag on and off.  

The summary – more tools around best practices around this thing which is becoming more mainstream.  With DevOps becoming more popular, more people are thinking “yes we need to get to production quicker, we need that cycle time to reduce” so it’s a natural extension I think to start solving some of those problems with feature flags.  

“I think it’s just starting to become more mainstream frankly because it’s a solution to a problem that is starting to become more mainstream.”   

29 Aug

Guide to Feature Flagging Best Practices

While there are many companies who practice feature flagging as a way to extend continuous delivery and release faster, there are just as many companies who are just starting out with the practice.  And until now, there hasn’t been a great resource for the feature flagging beginner.  Feature flagging as a concept is super straightforward, but must be implemented with diligence.  Haphazard feature flags can devastate a company, whereas well-managed flags can prevent disaster and even make your company more competitive.  


LaunchDarkly’s guide to feature flagging best practices addresses these deficiencies and provides a roadmap for better feature flag management.  While many aspects of feature flagging must be personalized to meet a particular company’s needs, the guide acquaints newcomers with how powerful feature flags can be and what they need to watch out for.  It also relays some pro tips for those who have been feature flagging for a while.  

LAUNCHDARKLY HELPS YOU BUILD BETTER SOFTWARE FASTER BY HELPING MANAGE FEATURE FLAGS AT SCALE. START YOUR FREE TRIAL NOW.

26 Aug

3 Ways to Avoid Technical Debt when Feature Flagging

Feature flags are a valuable technique of separating out release (deployment) from visibility. Feature flags allow a software organization to turn features on and off at a high level, as well as segment out their base to allow different users different levels of access. However, feature flags have an (ill-deserved) reputation of “Technical Debt”. Used incorrectly, feature flags can accumulate, add complexity, and even break your system. Used correctly, feature flags can help you move faster. Here’s three easy ways you can avoid technical debt when using feature flags.

  1. Create a central repository for feature flags

Using config files for feature flags is “the junk drawer” of technical debt. If you have seven config files with different flags for different parts of the system, it’s hard to know what flags exist, or how they interact. Have one place where you manage all of your feature flags.

  1. Avoid ambiguously named flags

Give your flags easy to understand, intuitive names. Assume that someone other than you and your flag could potentially be using this flag days, months, and years into the future. Don’t have a name that could cause someone to turn it on when they mean off, or vice versa. For example “FilterUser”, when it’s off – does this mean users are filtered? or not?

  1. Have a plan for flag removal

Some flags are meant for permanent control, for example for an entitlements system. Other flags are temporary, meant for the purpose of a release only. If a flag can be removed (because it’s serving 100% or 0% of traffic), it should be removed, as quickly as possible. To enforce this rigor, when you write the flag, also write the pull request to remove it. That way, when it’s time to remove the flag, it’s a two second task.

13 May

LaunchDarkly – Now Serving Billions of Feature Flags at Warp Speed

It’s been a busy spring for LaunchDarkly, the leading feature flag management company.  We’ve hit a huge milestone – we’re serving  Billions (with a B) feature flag events daily, for companies all over the world. 

LaunchDarkly helps companies with better DevOps by managing code deployment separately from business logic. We allow an entire product development organization to control who gets the right feature, when. Developer tool companies like CircleCI, Apiary and AppDirect use us to power their own DevOps.  But, LaunchDarkly isn’t just for developer tool companies to reduce risk and iterate quickly on feature release.  Across industries, companies like Ten-X, Behalf, Handshake, and TrustPilot integrated LaunchDarkly into their development and release cycles to move faster with higher quality. 

Why are companies moving so quickly to LaunchDarkly?  We offer true polyglot support for web and mobile applications. Have a Node.js web application with Go micro-services and a mobile app written in Swift?  We got you covered.  And, more importantly, we allow companies like CircleCI and Behalf to go beyond DevOps to transform the way they deliver software from idea to delivery by uniting product management, engineering, QA and operations. We offer end to end feature flag management to allow an entire product development organization to control who gets the right feature, when.

To help spread the word about the power of feature flagging, we also have awesome partners.  We were thrilled to participate in the Microsoft Build in March.  I was part of the Microsoft DevOps Keynote, showing our end to end integration with Visual Studio Team Services.  And we’re proud to power feature rollouts for effective DevOps at companies like: Continue reading “LaunchDarkly – Now Serving Billions of Feature Flags at Warp Speed” »

08 Dec

Feature Toggles at Tout

How Tout uses feature toggles in their development process, guest post by Javid Jamae, Principal Engineer (Growth at Tout).

Tout is a video platform for media brands and content publishers. We have video players on hundreds of prominent high-traffic media websites and blogs across the United States. Using our video marketplace, websites can show premium video content from producers such as Time, Yahoo, Condé  Nast, The Wall Street Journal, ESPN, and dozens of others. The sites can also let their journalists and authors capture their own video content using our Android and iOS apps, and then play their content on their sites.

The way our service works is that we provide a Javascript embed code that our partner sites place within their website template. When their page loads, we inject our video players into various parts of the page. Deciding everything from where the video players placed, what video content it has, and how it appears can be decided on the server side, thus minimizing configuration and code changes for our customers.

In the early days, we only had a few sites using our new products. To keep things simple, we kept most of our product configuration in the code and consistent across all of our partner websites. When we wanted to make a change, we would negotiate with all of them before rolling out a new feature. This was manageable because there was little to no variability and notifying  individual partners of changes was easy.

As our customer base grew, we found it increasingly difficult to maintain a single product configuration, so we introduced a way to configure products differently for each partner website. But, due to the nature of being an embedded product, we started finding it difficult to test changes internally and to demonstrate changes to our partners.

We did internal testing using a web debugging proxy software called Charles. We would deploy an updated version of our code to our staging (i.e. internal test) environment, then we would configure Charles to proxy our production server to our staging server on the machine we were testing on. Though there are a few snags with this process, it works relatively well overall. The problem with this approach is that it only works for internal testing.   

Our customers often wanted to preview new changes to their site before we rolled them out so that they could do their own verification and training. As the demand for customer previews increased, we started looking into feature toggles as a solution.

Conceptually, feature toggles are simple, you just keep the old version of your code in place, but wrap it in a conditional clause that does the new thing if the feature is enabled:

Since we are using Ruby on Rails, we started by using a Ruby library appropriately called ‘feature’. The feature library is an open source library hosted on Github. The great thing about this library is that it enabled us to create feature toggles in our code within minutes.

Using this library had several downsides as well. Since it’s just a library, it has no user interface for creating or modifying feature toggles. It also has no functionality for passing in feature toggles from web browsers using request parameters or cookies, which is tremendously useful for testing a specific state of your toggles. Another thing that is missing is metadata around when and who modified the state of the toggles, which is really useful for debugging and forensic purposes. Nevertheless, with a bit of work building out those tools, the feature library was great for getting up and running and we were able to deploy our code to our production with a new feature toggled “off” and then enable it for testing or demo purposes.

feature switch

After starting to use feature toggles, we realized several unintended benefits. First, if we found any issues with the new version of the code, we could merely roll back to the old version by switching a toggle. We no longer needed to do roll-back or roll-forward deployments to undo a mistake in production.

Another great benefit was that it minimized environment contention. Since our team is still relatively small, we only have one test environment. By using feature toggles, everybody could merge and deploy their changes to the staging environment and just toggle “on” the features that they want to test by setting cookies in their browser.

We also started using feature toggles as a way to do zero-downtime feature cutover. Since we’re using a microservice architecture, we have multiple systems that all talk to one another. Previously, if we needed to roll out a new feature that required communication between multiple systems, we would have to bring down all of the related systems and deploy all of the appropriate code changes before bringing the systems back up. But with feature toggles, we were able to deploy each system code change when it was ready and then just synchronize toggling the new functionality on. Effectively, this eliminated the need to have complex blue-green deployments since we just managed it all with feature toggles.

However, using toggles comes at a cost, though I feel like the cost is definitely justified. First, not all of our developers were thrilled about having temporary if/else statements littered throughout the code. They felt that this added extra complexity in the code and in the unit/integration testing effort. We started developing some good best practices around where to put the feature toggles and updated our development process to include the proper removal of features that had already been rolled out. While unit testing did become more complex, the overall complexity of our process was simplified. We no longer required as many unnecessary feature and merge branches and the deployment and process became simpler as I mentioned earlier.

All in all, I’d say that using feature toggles has been a great tool in our tool belt and has had significant positive benefits to our development process. Though we used an open source library and grew the tools that we needed around it, tools like LaunchDarkly (which didn’t exist when we got started) would quite possibly have gotten us to where we needed to be much faster by providing us with all the tools that we had to build from the ground up and the support that we didn’t have.