02 Apr 2018

Targeting Users with Feature Flags

Companies exploring feature management are looking for control over their releases. A common theme is using feature flags to rollout a feature to a small percentage of users, or quickly roll a feature back if it is not working properly. These future flaggers also seek to control a feature further by limiting its visibility to individual users or a group of users. In this piece, we’ll explore how LaunchDarkly lets you control your releases in these ways.

When launching a new feature, or simply updating an existing feature, you may not always want everyone to receive the same experience. When working with a new front-end marketing design, a back-end improvement to search, or anything in between, targeting specific users can help ensure the best experience for all of your customers. Below are a few common scenarios where targeting can come into play. We’d love to hear your thoughts on other targeting best practices—let us know what you come up with!

Targeting users within LaunchDarkly’s UI.

Internal QA

Perhaps the most common use case for individual user targeting is internal QA. Testing in production is a scary thought, but so is launching to production without knowing how the rollout will go. Feature flags enable safe testing in production. By targeting only your internal QA teams to receive the new feature, you can experience how it will function in a production environment without exposing the untested feature to your customers.

Beta Releases

When releasing a feature or product in beta, you can target users in your beta group to receive this update while not releasing it to anyone else. Essentially, you would set the flag to “true” for all beta users to enable the new feature. Everyone else would have the flag set to “false”. When Kevin from LaunchDarkly asks to be part of your beta group, it’s as easy as adding that user to the true variation to enable the new feature.

Attribute-Based Targeting

Sometimes, you’ll want to target users, but may not want to dive into as many specifics as we just visited. Perhaps you are constrained by regulations, and need to deliver a different experience to customers from different states. Or, maybe your marketing team is launching a campaign specifically to Gmail users. You can place the email domain “gmail.com” in your targeting rules to deliver an experience only to that group.

Targeting in a Canary Launch

Many companies are seeking to slowly rollout their releases to before releasing to everybody. Rolling a new feature out to 10% of users, for example, is a great way to gain validation before a complete launch. You can target your internal QA team to be a part of the 10% initial rollout, or specifically include members from a beta group to see the new feature first. Conversely, you can target specific users or groups to exclude from the launch, and give them the new feature only when it is rolled out to everybody.

A common challenge with canary launches is delivering a consistent experience to users from the same group, especially in B2B applications. LaunchDarkly’s platform provides a bucketing feature that solves this problem. By bucketing users, you are ensuring that all users from the same group receive the same variation for the duration of a canary launch.

Other Considerations

The examples we have covered so far are typically used for short-term flags, either upon initial rollout or for a shorter-lived feature. As your company’s feature flag use continues to mature, wrapping a feature or set of features in a long-term or permanent flag is a logical next step. A common use case here applies to entitlements. For example, if you have Basic, Pro, and Elite versions of your product, user targeting helps manage these tiers effectively across your organization. Many products with different account tiers already have a way of controlling functionality, but with a feature management platform, this can be easily managed by anyone on your team. Customer success team members can quickly dial functionality up or down without the need to go to engineering for support. When a customer upgrades from Pro to Elite, a user simply flips the appropriate flags on.

Managing targeting rules at scale is an important consideration, especially as the amount of feature flags you are managing continues to grow. We introduced a new feature we call Segments to help tackle this very topic. Segments enables you to build reusable targeting rules to make this practice quick and scalable. Borrowing from a few of my examples above, you may want to flip a feature on for everyone with an Elite subscription, but also include all beta members as well. Because this is an important feature, you may want to also target internal users to ensure rollouts go smoothly. This same rollout strategy will likely apply to future features and you can simply target your Segment to receive the true or false variation. We’ll be featuring Segments in a future post, so stay tuned for more.

Build reusable targeting rules with Segments

Data Privacy and Targeting

Targeting often involves customer data, and we launched Private Attributes in January to add an additional layer of protection. Private Attributes allows you to shield your customer data from LaunchDarkly while still enabling targeting rules on the shielded data. One thing I’d like to point out is that you never need to upload a database of users into LaunchDarkly to enable targeting. LaunchDarkly simply stores your targeting rules, which are evaluated based on the user objects that you provide. Ben Nadel wrote a wonderful piece describing InVision’s use of targeting and Private Attributes: Using LaunchDarkly To Target Personally Identifiable Information (PII) During Feature Flag Evaluation Without Leaking Sensitive Data.

To wrap things up, there are a number of different ways targeting users can deliver a more valuable experience. We visited just a few above, but the opportunities here are endless. At the end of the day, you’ll target users based on what makes the most sense for your product. We’d love to hear your thoughts and how you think user targeting can help. Until then and for more pointers, you can visit our documentation here.

01 Mar 2018

Privacy Shield and GDPR

In today’s age, where your data goes and how it is used is a new, clear, and present danger that we all face. At LaunchDarkly, we believe that the your information is yours and not ours, and the information that you have about your customers should never be used in a way that they or you did not intend. We’re pleased to announce that LaunchDarkly is amending and expanding our Privacy Policy to be US Privacy Shield and General Data Protection Regulation (GDPR) Compliant.

I’m Tim Wong, the Data Protection Officer here at LaunchDarkly, and I would like to tell you about what how things are changing. First of all, we’re making the commitment to never sell data in LaunchDarkly to third parties, so long as LaunchDarkly exists. Our customers pass their information to us in confidence and the Service Data that we collect will stay that way.

Secondly, we adhere to all of the guiding principles of the GDPR that will go live in the end of May. That includes the right to be forgotten, the right to know what data we have, and to not process data without consent. We submitted to having an independent third party audit our security/data protection/audit capabilities.

We’re also committing to creating a trust site that tries to be clear about where we stand in plain (as we can have) English and to provide a clear English changelog should our policies ever change.

We understand that data, privacy, and GDPR is important to our customers and we look forward to partnering together moving forward. If you have any concerns, questions, and especially cookies to forward us (not the digital kind), please email me at privacy@launchdarkly.com

10 Jan 2018

Launched: Private User Attributes

Today we are launching a new feature called Private User Attributes. This feature allows our customers to collect event data from feature flag evaluation while controlling which user attribute fields are stored or exposed in other parts of the LaunchDarkly user interface.

In our feature management platform we use the stream of flag impression events generated by our SDKs to power many useful features—including attribute autocomplete, our users page, the flag status dashboard and A/B testing. Some of our customers have requirements to restrict Personally Identifiable Information (PII) from being sent to LaunchDarkly. Until now, these customer’s only choice had been to completely disable any event data that might contain PII fields (emails, user names, etc.).

The introduction of Private User Attributes means all customers now have the the ability to selectively choose not to send back some or all user attributes. With this selective approach, customers can continue to view flag statuses, use A/B testing, and view and override attributes for targeted users while protecting sensitive data. We have also made several small changes to the LaunchDarkly web UI to display when attributes are not available and offer reasonable choices when data has been marked as private and is unavailable for display.

What is a Private User Attribute?

When a user attribute is declared private it can continue to be used for targeting but its value will not be sent back to LaunchDarkly. For example, consider the following targeting rule:

if e-mail endsWith "@launchdarkly.com" serve true

To implement this in your application, you’d make this call to LaunchDarkly:

ldclient.variation("alternate.page", {
  key: "35935",
  email: "bob@launchdarkly.com"
}, false)

When this code is executed, a flag impression event would be sent back to LaunchDarkly containing the following data:

  {
    "key""alternate.page",
    "kind""feature",
    "user": {
      "email":"bob@launchdarkly.com",
      "key""35935"
    },
    "value"true
  }

In the example above, LaunchDarkly has received the email address of a user. Suppose now that you want to target by email address but not send that personally identifiable address back to LaunchDarkly.

This is what private attributes are for—you can mark the email field ‘private’. If your application marks the email field private, you can still target by email address but LaunchDarkly will receive the following data:

  {
    "key""alternate.page",
    "kind""feature",
    "user": {
      "key""35935",
      "privateAttrs": ["email"]
    },
    "value"true
  }

This data object no longer contains the targeted user’s email address. Instead, there is a record that an email attribute was provided during feature flag evaluation but that its value has been excluded from the event data.

Ok, let’s hide some stuff.

There are three different ways to configure private attributes in our SDKs:

  • You can mark all attributes private globally in your LDClient configuration object.
  • You can mark specific attributes private by name globally in your LdClient configuration object.
  • You can also mark specific attributes private by name for individual users when you construct LDUser objects.

It should be noted that the user key attribute cannot be marked private, for this reason we encourage the best practice of not using any PII as the user key. For more in-depth docs you can look at the SDK Reference Guides.

How do private attributes work for mobile and browser SDKs?

You might recall that our mobile and browser SDKs work slightly differently—flags are evaluated on LaunchDarkly’s servers, not in the SDKs. We still support private user attributes for these SDKs; when evaluation happens, we do not record any data about the user being evaluated. For analytics events, these SDKs strip out private user attributes the same way our server-side SDKs do, so your data is still kept private.

How will marking attributes private affect my experience?

At LaunchDarkly usability is critically important to us (after all, we use LaunchDarkly to build LaunchDarkly). In thinking through the impact of reducing the amount of data that is visible we wanted to still provide an intuitive UX for existing features. Because we record the name—but not the value—of a private attribute, name autocomplete will continue to function for private attributes in the UI, but autocomplete will not be offered for attribute values. We also indicate which attributes on a user are marked private on the User Page and when we are unable to predict a user’s Flag settings because a targeting rule uses a private attribute. Below is an example of the User page for a user with attributes marked as private:

Why shouldn’t I just make ALL my attributes private?

Making all your attributes private would limit some of the product features that rely on having attribute values available. For example:

  • The UI will not offer to autocomplete values. You’ll be able to see the names of hidden attributes, but not their values. So you’ll know that (e.g.) “email” is available for targeting, but you won’t know (in the UI) what email addresses exist.
  • Flag settings (on the users page) won’t be exact. We’ll be able to know if the user has a specific setting enabled, but we won’t be to predict what variation a user is receiving because we won’t be able to evaluate the targeting rules when hidden attributes are in play.

Given these limitations, our recommendation is that you selectively use Private User Attributes only for the purpose of securing PII.

We’d like to thank all the customers that provided feedback on this feature, we hope you enjoy not sharing your data.

30 Nov 2017

Launched: Personal Access Tokens

LaunchDarkly users can now create personal API access tokens. Anyone who uses our API in an automated fashion knows they must authenticate with an access token. Until recently, this meant there was one token for an entire organization or account. To meet the needs of larger teams as well as others leveraging the LaunchDarkly API we wanted to provide not only greater granularity in responsibility but also better accountability. In our original token model only admins had the ability to create and view tokens since they were afforded global access and control.

What exactly are they?

Personal access tokens are an improvement on this process. Now every user can create their own access tokens for authenticated API access. Users can scope permissions, making it easier to create different tokens for different use cases. So, if API access tied to a particular account should only have read access, now you can scope the token appropriately. Also, built-in and custom roles can be assigned to access tokens in the same way that you give a team member a role. This means that the roles-based controls you put in place for the web application are extended fully to the API access on our platform as well.

Best Practices

Now that personal access tokens are available, we recommend LaunchDarkly users migrate to using these instead of account-wide tokens. The old tokens will continue to function, but we also recommend deleting those for improved security.

In terms of using the personal access tokens, we strongly suggest creating a new token for each use case and giving them the smallest number of permissions necessary to do its job. We are proponents of the principle of least privilege and security best practices that limit your risk. After all, our goal is to help you eliminate risk, not increase it.

With personal access tokens teams can also more effectively manage accountability. With an audit log, admin no longer need to worry about figuring out who has done what with a single token. Now they can monitor all access tokens based on who created them. Tokens can never do more than the user who created them, and if the user’s permissions are downgraded, so are the token’s. When the only constant in modern infrastructure is change, it’s important to have visibility into who has done what and a durable record of when it was done.

You can read more about personal access tokens in the documentation here. And if you have any comments, you can contact us here, or make comments directly in the repository.

09 Nov 2017

Measure Twice, Launch Once

You want all your developers to have access to the main trunk of code to deploy — that’s the point of trunk-based development. It’s important they can put their code out as often as they want and iterate on their projects. However, you don’t always want developers turning on features that will have customer impact without some way to reverse course.

Secured activation is an under-appreciated part of feature management. Your developers can deploy code whenever they want—but when it comes time to test it externally, or turn it on for everyone, you can use settings to make sure that only a select group of people has the permissions to do so. All the activation changes should be tracked and audited to ensure that all activations have an accountability chain.

At LaunchDarkly, we have found that it’s good to be permissive about who can use and create feature flags, and restrictive about who can activate them. If you are trying to get started with transitioning to using feature flags more broadly, you might want to think about how to implement a repeatable process. You might also want to leverage LaunchDarkly’s ‘Tags’ feature to help with the organization and custom roles to assist with delegation and access.

You want the following qualities in people who have the permissions to change your user experience:

  • Understands the business reason for making the change
  • Has the technical knowledge or advisors to know when the code is ready to go live
  • Has a process in place for making the change and then testing it

You don’t want to have only one person who can do this, because they’ll inevitably become a bottleneck. Make sure your process can keep releasing even if a key team member is unavailable.

In the beginning you may look to put a process around every change and then look for optimization in that process. However, over time you should look into determining  what level of change merits process and what can be executed more easily. In some cases this might even allow for small changes to be approved or executed by individual engineers. Usually, features that have anything to do with money, user data collection, or changes in the user process should have a formal approval process. Changes to backend operations can be quieter and therefore need less formal process and lean more heavily on automated testing and peer review.

Think about your current deployment process. What happens if someone releases something too early? How do you protect against that? How will you port that control over to the access control that LaunchDarkly offers? What is the failure case if something doesn’t launch properly?

Feature flags are easy to implement in code, but managing them well across an organization takes some planning and forethought.

20 Oct 2017

Keeping Client-Side Feature Flags Secure

LaunchDarkly’s JavaScript SDK allows you to access feature flags in your client-side JavaScript. If you’re familiar with our server-side SDKs, setup for our JS SDK looks superficially similar—you can install the SDK via npm, yarn, or via CDN and be up and running with a few lines of code. However, if you dig a bit deeper into the source code and do an apples-to-apples comparison with our node SDK, you’ll see that our client-side SDK works completely differently under the hood.

There’s one simple reason for this difference—security. Because browsers are an untrusted environment, our client-side SDK uses a completely different approach that allows us to serve feature flags to your users securely. While we’ve made our approach as secure as possible out of the box, there are three best practices that we recommend all companies follow to maximize security when using client-side feature flags.

1. Choose Appropriate Use Cases

First, in a browser environment your entire JavaScript source code is fully exposed to the public. It may be minified and obfuscated, but obfuscation is not security. This means that if you’re using client-side feature flags, you must be aware that anyone can modify your source code directly and bypass your feature flags. For example, if you have a code snippet like this:

You can expect that any of your end users could inline edit the above code snippet in their browser window to show the brand new feature, bypassing the feature flag and its rollout rules entirely. This is a fact about how client-side feature flags work—not a limitation in the LaunchDarkly platform.

This doesn’t mean that client-side feature flags are always a security vulnerability—it just means that there are some use cases where a client-side feature flag alone isn’t appropriate. For example:

  • Launching a feature that is press-, publicity-, or competitor- sensitive. In this situation, having a user enable the feature early could derail a launch, or tip your company’s hand. Even the feature flag key itself (e.g. brand.new.feature) could be sensitive. In this scenario, you can’t even afford to push the new front-end code to production prior to launch, even if it’s kept dark.
  • Entitlements. A client-side feature flag alone is not a sufficient control for locking users out of functionality that they shouldn’t be able to access. In this case, a client-side feature flag is fine assuming it’s coupled to a backend feature flag. For example, you can use the same feature flag (with the same rollout rules) on both the front-end (to disable a UI element) and the back-end (to control access to a REST endpoint triggered by that same UI element).

Using client-side feature flags securely starts with choosing appropriate use cases. Ultimately, the choice depends on whether or not you can expose your new feature’s code to users that may not have the feature enabled.

2. Enable Secure Mode to Protect Customer Data

There’s a corollary to the observation that all JS code is fully exposed to the public—there is no way to securely pass authentication credentials to a 3rd party API from the front-end alone. Because anyone can view source, any API key or token that you pass to a 3rd party API via client-side JS is immediately exposed to users.

One solution to this problem is to sign requests—use a shared secret on the server-side to sign the data being passed to the 3rd party API to ensure that the data hasn’t been tampered with, or the request hasn’t been forged. Tools like Intercom take this approach. In Intercom’s case, request signing guarantees that a user can’t fetch chat conversations for a different user.

LaunchDarkly follows this approach as well. We call this Secure Mode, and when enabled, it guarantees that a user can’t impersonate another user and fetch their feature flag settings. Our server-side SDKs work in tandem with our client-side JavaScript SDK to make Secure Mode easy to implement. We recommend that all customers enable Secure Mode in their production environments.

3. Use a Package Manager to Bundle the SDK

There’s another aspect of security that comes with using any 3rd party service in your JavaScript—protecting yourself in case that service is attacked. Our recommended best practice is to embed our SDK in your code using a package manager, rather than injecting the SDK via script tag. We offer easy installation with popular package managers like npm, yarn, and bower. This does put the onus on you to upgrade when we release new versions of our SDK, but at the same time it can reduce latency to fetch the SDK and give you control in accepting updates.

Conclusion

We go to great lengths to ensure the security of our customers and their end users. We follow industry best practices in the operation of our service (we recently completed our SOC2 certification) in addition to providing advanced security controls in our product such as multi-factor authentication, scoped API access tokens, and role-based access controls. This approach to security extends to our client-side JavaScript SDK. By following the practices outlined here, you can ensure that your use of client-side feature flags keeps your data and your customers’ data safe.