16 Feb

Net Promoter Score + Feature Flags for canary releases

Canary releases are a DevOps best practice of pushing new features to a subset of your customers. By pushing to a subset, this group can provide early feedback, verify functionality, and act as “canaries” to your entire population. Originally, canary had a negative connotation – miners took canaries down into mines, as canaries were more sensitive to bad air. If the canary stopped singing, the miners knew it was time to skedaddle.

Surprisingly, being a canary can actually make your customers even bigger fans! I heard a great use case from a LaunchDarkly customer People.ai,  AI for managing sales teams, who would automatically push new features out to users who had an NPS of over a certain threshold. Net Promoter Score (NPS) rates how strongly would a customer recommend your solution, so initially, this seemed bizarre.  Why risk showing your best advocates functionality that might be unstable, early, or fragile? However, the people who are happiest users (likely to recommend) are actually very excited about being the first to see new functionality. They love feeling plugged-in to new development. And in addition, with LaunchDarkly feature flag management, if a feature isn’t doing well, the team can instantly turn it off, reverting to the old, stable experience.

I love hearing stories about how customers are using LaunchDarkly’s segmentation capabilities to release better software, quicker.

 

26 Jan

Launched: Flag Tagging Management

LaunchDarkly Feature Flag / Feature Toggle Targeting and Management

For better feature flag management, LaunchDarkly allows you to create tags for organizing and grouping your feature flags.  Adding tags (like “Front-End”, “Ops”, “Marketing”, “Restricted”) helps you categorize flags and manage custom permissions.

Here, we have added the tags “mobile”, “marketing”, and “unrestricted” on the Settings tab of our feature flag:

After adding tags, you can filter by tag on the dashboard and link to filters for better feature flag management.

Here, we have clicked on the “marketing” tag, which has created a filter that shows all feature flags tagged “marketing.”

LaunchDarkly Feature Flag Tagging and Management Dashboard

Creating a filter also generates a URL that you can bookmark and share with your teammates.  For example, this URL will show all feature flags tagged “marketing”  https://app.launchdarkly.com/default/production/features?tag=marketing .

In the future, we will add more advanced filtering and sorting for even better flag management.  If you have any suggestions or questions, please feel free to contact us at support@launchdarkly.com

23 Jan

Controlling Releases with Microsoft VSTS + LaunchDarkly

LaunchDarkly and Microsoft controlled rollouts and use targeting feature toggles and feature flags

It’s every project manager’s dream to get complete control over their software releases and have a way to continuously integrate and deliver new features, while also controlling the visibility of those features once they’re live.

Microsoft Visual Studio Team Services

Microsoft’s Visual Studio Team Services (VSTS) enables development teams to build, integrate, test, and release new software in a single platform. It has a centralized version control system that allows for continuous integration, package management, and release management. Together, these components enable agile teams to move faster in a more centralized manner while also minimizing the number of third party dependencies needed for deployment.

Traditionally, when you release new features into your Production environment, they are live for all of your users. You could provide code-level access control via a config file or modify database values to superficially control the visibility of your new features. However, these release controls are engineer-dependent, meaning they require a developer to make changes to who gets to see a feature and when. It makes it very difficult to target granular segments of users or perform incremental percentage rollouts to test performance and visibility.

LaunchDarkly VSTS Extension

With LaunchDarkly, you can use feature flags to control the full release lifecycles of your features, perform percentage rollouts, and granularly target user segments. These controls are surfaced via a UI that can be used by non-developers, including designers, project managers, and the business team.

LaunchDarkly and Microsoft Visual Studio Team Services (VSTS) Feature Flags / Toggles for release management

To tie this feature flag UI into your VSTS workflow, the LaunchDarkly Extension  allows you to associate feature flag rollouts with VSTS work items to get complete control over who sees what, and when. This allows teams to:

  • Centralize feature and release lifecycle management
  • Release confidently with less risk
  • Incorporate feature flags into the release process
  • Team support for superior project management
  • Gain better visibility into features and outcomes
  • Achieve next-level continuous delivery
  • Empower your team to do better DevOps

Here’s some of the core functionality when you integrate VSTS and LaunchDarkly:

Associate feature flags with work items

Take full control of the release lifecycle of your work items and manage feature rollout

LaunchDarkly and Microsoft Visual Studio Team Services (VSTS) Feature Flags / Toggles for release management work items

Comprehensive Release Management

Manage percentage rollouts and turn the feature flag on or off

LaunchDarkly and Microsoft Visual Studio Team Services (VSTS) Feature Flags / Toggles for release management controlled percentage rollouts

 

Incorporate feature flags into your release definitions

Tie feature flags to your Visual Studio Team Services release definitions and perform percentage rollouts:

LaunchDarkly and Microsoft Visual Studio Team Services (VSTS) Feature Flags / Toggles for release management work items and rollouts

Getting Started

If you already use VSTS and have a LaunchDarkly account, then all you have to do is connect the LaunchDarkly extension and you’re ready to feature flag and take total control over your releases.

You can also check out the documentation to learn more about setup and integration.

04 Jan

Running Usability Tests in Production

Usability testing in a real-world environment (aka production environment) gives us insight into how users actually use our product in their day to day lives. It is one thing to run a test in a lab setting, but it is another to have users try features while they are walking, running to the airport, stressed, and sleepy.

But, no matter how hard we try, it is very difficult to truly simulate how our apps behave in our production environment. We can run focus groups, beta tests on a beta environment, and test things internally, but how can we truly mimic the real world in an artificial context? In other words, how do we test a feature while also simulating the environment it’s meant to be used in?

LaunchDarkly Usability Testing In Production - Feature Flags and Feature Toggles - Context

A good real-world usability test analyzes features efficiently and accurately collects user feedback to improve the user experience. But, when we test anything in a non-production environment, we are inherently biasing our tests. In a lab-based usability test:

  1. Users are overly cognizant of the feature they are testing.
  2. Users are unnaturally focused on testing that new feature.
  3. Users are using fake data or incomplete data, and don’t sufficiently utilize actual use cases.
  4. It is very hard to test discoverability (i.e can a user find the feature on their own?).
  5. Users tend to ignore distractions, like external notifications (Facebook, Skype, texts) and just focus on the task at hand.
  6. Users are in an overly analytical mode, typically looking to give feedback.
  7. Users are overly forgiving, looking to please!

Does this mean that running usability tests in non-production environments is a waste of time? Not at all! This is absolutely necessary to identify bugs, check for general usability, and solicit feedback quickly.

However, it should just be one of the steps in a comprehensive usability testing process, one that involves internal, staging, and production usability testing.

Benefits of usability testing in production:

The primary purpose of usability testing in production is to gather real-world user behavior while minimizing bias and performance risk. Some more benefits include:

  1. Genuine user feedback in a real-world environment
    • Quantitative insight into your feature’s performance. How well is it scaling? Are users using it? How is it impacting your system? How are your levels of engagement?
    • Contextual insight into your feature’s efficacy. Do users see the new feature? How is it meshing within the context on your existing feature set? Are people using it as intended? Are people using it once and then not using it again?
    • Qualitative feedback. Are users complaining? Are they happy? Are they neutral?
  2. No opt-in bias – users test the feature without knowing they are part of the test. You can assess how well they use it by using a product like Full Story to record the session or by tracking metrics. You therefore get a more representative sample testing the feature, rather than just early adopters.
  3. Measuring actual system performance – there is nothing quite like your production environment, where you can have a complex array of nodes, clusters, CDNs, etc allowing your app to scale. As people start to use the new feature, you can see how it is impacting your actual system (load times, discoverability, caching issues).

Managing a usability test in production:

Of course, testing anything in your production environment is inherently risky and has real-world consequences. If you launch something to everyone just to get feedback and they hate it, then you risk permanently losing those users. Equally bad, you can cripple your entire application with unforeseen scaling and performance issues.

To mitigate this, companies like Facebook, Amazon, and Google collect production feedback by releasing features behind feature flags. While we won’t go into to the specific anatomy of a feature flag, we can go through the methodology behind the release.

LaunchDarkly - Usability Testing Using Feature Flags and Toggles - Betas and Feedback

If a feature is wrapped in a feature flag, then it gives you control over who sees the feature and when. This means that you can perform targeted and controlled releases using a percentage rollout, whereby you can incrementally increase a feature’s visibility to 1%, 5%, and to 100% of your users.

Hence, you can collect production level feedback because you control the level of risk. If a new feature is performing well, then you can keep increasing the percentage rollout. If it is tanking or hurting performance, you can reduce the rollout or kill it completely.

Therefore, feature flags (aka feature toggles) give you full control over the risk of your production releases. You can gather real-world user feedback by separating your feature rollout from your code deployment.

22 Dec

A New Way To Beta Test

LaunchDarkly Feature Toggle Beta

It’s best practice for products to have some sort of beta – a way to collect customer feedback and test performance before releasing to everyone. In an era of continuous delivery, we are delivering new features and experiences more frequently and with less time to gather thorough customer and performance feedback. With this increased cadence, product teams are having to make betas shorter, forego them altogether, or slow down their release cadence to gather adequate customer feedback.

Challenges of traditional betas:

  • Coordinating Opt-Ins: It sometimes takes weeks or months to gather customer opt-ins to test new betas. You also have to organize the distribution of beta keys (ex. for early access to games) and reminder emails.
  • Organizing Focus Groups: Getting feedback from focus groups is often time consuming and expensive, creating a long feedback loop that lengthens the release process.
  • Opt-Out: If customers opt-in to a beta and don’t like the experience, then they will want a simple mechanism to switch back to the production version.
  • Granular Betas: It is very difficult to do targeted betas based on user attributes or to perform incremental percentage rollouts of new beta features.

Feature toggles

To overcome these challenges, smart product teams are beginning to run betas with feature flags/toggles. These are mechanisms for granularly controlling software releases, allowing you to control the timing and visibility of a beta release.

Currently, many betas are tied to code releases and are managed by a config file or database.  This approach requires engineering time or custom mechanisms to opt-in users.

With feature toggles, you can empower product, marketing, sales, and even customers (themselves) to opt-in new to a new beta experience.

Feature Toggle Beta Test LaunchDarkly

In this simple example, you can use a toggle to control the visibility of a new beta feature. Ideally, this toggle would be part of a user interface that could be controlled by a non-technical team member. The code, itself, could be deployed off and then turned on via the toggle.

Beta Test Percentage Rollout with Feature Toggle LaunchDarkly

Moreover, you can also use the toggle to control the percentage of users who get the beta experience. For instance, you could release the new beta experience and have it rolled out to 0% of users. You could gradually increase the rollout percentage from 1% to 5% to 20% and more, collecting customer and performance feedback along the way.

Surfacing this beta control functionality in a user interface is critical for giving non-technical team members access to release controls.

Regional betas

For a recent example of a targeted rollout, we can look at how Pokémon GO released their product country by country: first to the United States and then abroad.

This is a great use case for feature toggles because you can create targeting rules to determine which users receive the feature first. For example, I could create a toggle that is governed by the rule: “If users live in San Francisco, then serve the new Nearby Pokémon feature”. This allows you to maintain different regional feature sets without having to deploy different versions of the application. It also allowed Pokémon GO to refine their algorithms and assess customer feedback before rolling out the new feature to a wider audience.

Benefits of beta testing with feature toggles

  • Empowered non-technical users: Allow the sales, marketing, product, design, and business teams to turn features on for specific users, collect feedback, and control the business logic. This also substantially cuts down on engineering time.
  • Production feedback for your beta tests: Test features in production with limited user segments to collect customer and performance feedback.
  • Incremental percentage rollouts: Gradually roll out features to incrementally test performance and mitigate risk. If the feature is bad, toggle it off.
  • Real-time opt-in / opt-out: Allow users to opt in and out of beta tests in real time, controlled via the feature toggle. Skylight provides a nice article on this.

Getting started with toggles

Conceptually, a feature toggle is relatively simple. You create a conditional in your code that controls the visibility of a code snippet. There are many open source libraries that will allow you to get started.  However, these libraries become cumbersome when you try to feature toggle at scale or restrict access to particular toggles. Depending on your needs, you could consider a feature toggle management platform to provide a system for access control and mitigating technical debt.

06 Dec

Feature Flag-Driven Products

LaunchDarkly Consistent Mobile and Web Experiences using Feature Flags / Toggles

Using feature flags for plan management, personalization, and cross-platform consistency

When feature flags/toggles were first introduced, their primary purpose was to mitigate risk and manage software releases.  We would use these toggles to turn features “on” or “off”, and gradually roll out new features to customers.  If something went wrong or if customers didn’t like the new feature, we could kill it without redeploying.

What developers started realizing is that we could use feature flags to manage dynamic content and have long-term control over every application feature.  This means that toggles would not just be “on” or “off”, “true” or “false.”  Rather, they could serve strings, numbers, JSON objects, and JSON arrays — allowing us to serve dynamic content and use feature flags to perform more interesting functions.

 

Plan Management

Almost every product has a series of plans made of bundled features.  Plans typically have two parameters: a feature and a value.  A feature can be something like a VIP Area, enhanced support, or additional customizations.  Values are attributes of features, and can be something like “20” members, “5” teams, and “enhanced” speed.

Using feature flags, we can easily determine which features belong to which plans, and edit bundles as we add new features or change their values.

Here, we have two plans: Standard and Gold.  The Standard plan allows for 10 members, while the Gold plan allows for 50 team members and a VIP area.

LaunchDarkly Plan Management with Feature Flags and Toggles

We constructed two flags: team_count and vip_area.   For team_count, we created a multivariate feature flag that returns the numbers 50, 10, and 1.  We then have a rule that checks if the “plan” attribute matches “standard” or “gold” to determine which value to serve.  The “plan” attribute would belong to a user object, which could look something like this:

Because Ernestina’s plan attribute has a value of “gold”, she would receive a 50 team member limit and have access to the VIP area.

Moreover, we could use multivariate flags to dynamically modify pricing.  For example, we could toggle prices on-load based on whether the the user is a “VIP” or “NEW”.

LaunchDarkly Pricing Management with Feature Flags / Toggles

In this example, all “new” users receive a discounted price of $50 and “VIP” users see a discounted price of $20, while all other users receive the normal $75 price.

Other pricing use cases would be localization, seasonality, and discount codes.  We could manage these separate from our application logic, giving us more flexibility without having to redeploy.

 

Personalization & Styling

We can also use feature flags to manage styling and personalized themes, essentially using these toggles as a quasi-content management system.

Here, we use a multivariate flag to serve color hex values, turning a feature orange, blue, or green:

LaunchDarkly Multivariate Feature Flags / Feature Toggles for dynamic application managementWe could also serve pre-defined CSS class names to add or remove properties from elements.  This allows us to separate some of this logic from our primary codebase.  While this shouldn’t be used for primary layout and styling, it could be used to serve particular to styles or to customize themes based on user attributes.

For example, if I had a user with a ‘theme’ attribute that has a value of ‘blue’, then I could serve the blue CSS class, which my application logic would append to the relevant elements.

 

Consistent Cross-Platform Experiences

Cross-platform feature flagging is an easy way to deliver personalized and consistent user experiences across different platforms.  If our product has features that are shared on a web and mobile platforms, we can use one flag to control the visibility of both those features.

Here, we have a user who is receiving the TRUE variation for the VIP feature flag.

LaunchDarkly Consistent Mobile and Web Experiences using Feature Flags / Toggles

Even though the web and mobile platforms have different permutations of the feature, the same flag could synchronize the VIP experience between both platforms.

Some more benefits of cross-platform feature flagging include:

  • The ability to decide whether to release a cross-platform feature simultaneously or separately, with full control over who gets to see that feature. For example, web users might get access to a new search bar before mobile users do.
  • Real time personalization that allows users to opt-in to new features on one platform (like mobile) and have that personalization sync with another platform (like web)
  • Percentage rollouts that allow us to gradually release a feature to targeted users on different platforms, allowing us to assess user and performance feedback for each platform
  • A kill switch that lets us turn off poorly performing features for web and mobile, without having to redeploy

 

Flag Management

The process of feature flagging is fairly straightforward: we wrap our features in conditionals that determine who can see the features and when. At an enterprise scale, organizations must confront the complexities of mitigating technical debt, managing developer workflows, compliance, and controlling the lifecycle of feature flags.

One of the biggest pain points around traditional feature toggles is that marketing or business teams would need to create engineering tickets to get their requests updated.  These toggles would be typically controlled within the codebase itself, like a config file or database.

By adding a user interface for flag management, we can empower non-technical teammates with the ability to target users, perform rollouts, or run beta tests.  This even makes it easy for engineers to coordinate flag creation, mitigate technical debt, and manage the lifecycle of their flags.

LaunchDarkly Feature Flag / Toggle Management Dashboard

Some other benefits of UI flag management:

  • Flag Implementation & Consistency – Feature flags need to be carefully crafted and consistently implemented within our application to prevent performance degradation and ensure that they function correctly. Often times, organizations will use conflicting methods like managing multiple config files or using in-line toggling.
  • Flag Scalability – Adding more feature flags makes testing and management exponentially more difficult over time. It becomes very hard to tell which flags are necessary or obsolete.
  • Flag Management – Maintaining feature flags across multiple development environments (local, QA, staging, production) becomes arduous and time-consuming. It becomes increasingly difficult to track who created the flag, the flag’s intended use, and changes made to the flag’s rollout.