21 Dec 2017

Toggle Talk with Happy Co’s VP of Product Phill Claxton

Phill Claxton has implemented feature flags in three different organizations. As a driving force on the business side, he’s seen his company evolve and witnessed firsthand the difference feature management has made in the overall progress of the company. When it comes to the most fundamental benefits of feature flagging, he points to continuous delivery and the speed of learning. Phill is VP of Product at HappyCo, COO at Storekat.com and a cross-functional consultant for early startups.

How long have you been feature flagging?

I’ve been a fanboy of feature flags for a little while. They’re a strange thing to love but it’s the result of implementing them that has made such a huge difference in my life, and the companies I’ve been involved with.

I think I can trace my first exposure back to a HubSpot article in 2015. We were faced with a challenge at a startup that I founded down here in New Zealand called DeskDirector. We had maybe 150 – 200 customers/accounts at the time, and we were hitting a challenge that’s a telltale sign for those on the other side of learning about feature flags. We had a very small development team, and it was very hard for us to do multiple things. We needed to decide what to build next and measure the impact from it.

We had made some pretty substantial failures, it’d be fair to say. We’d launch features that we thought were going to help the many but when they failed we realized it was the false wisdom of the few.

“So we were trying to work out a way to learn faster.”

At the same time we were thinking about how we can restructure our pricing, and it sort of came together when we read that Hubspot article. We realized we needed to control the release of features we were working on to cohorts of customers. That way we could quickly decide whether this had the broader appeal to launch to our greater market.

When do you think feature flagging is most useful?

Once you hit that 100 customer mark there’s some natural segmentation that lend itself to feature flagging to test out cohorts. The customer grouping starts to look about the same as it will at 1,000 customers. I would suggest that’s probably when you want to start really thinking about feature flagging.

I’d caveat that by saying we were in the B2B space. You might get there much sooner if you were selling a product to consumers.

“That being said—the last times we’ve done it, it was always too late. We could have started earlier. I suggest you start learning really early without worrying too much about cohorts.”

For me it’s always been two things, which is about continuous delivery and the speed of learning. Specifically, allowing a product team or engineering team, to launch something and learn at a much faster rate from the customer base. The same thing goes for pricing inside your product.

In the SaaS world, the need to experiment with pricing is a part of your standard business practice. And the earlier you can do that, the better. Sometimes people forget that there’s many different use cases for feature flagging.

Are there any cases where feature flagging is not a good idea?

Our practice was to flag anything that was impacting on the user’s experience. So, if the impact on the end-user experience is low and there’s a very high level of confidence in the change, then we’re comfortable not flagging. If the change itself is completely hidden from the users’ experience and service side, then that’s not normally something that we would need to consider feature flagging.

“I would say more often than not there are situations where it isn’t thought of enough. The default position should be that it is flagged.”

Best use of a feature flag—a personal story?

By this stage we were feeling mature in our development cycle and had flagging in place. It was a documentation platform that tracked expirations of devices that an IT company was looking after.

We were working on a new notification engine which would trigger expiration alerts to customers. And so we did feature flag this feature, and we’d gone through a lot of initial wireframes with customers. We’d even built out an early stage MVP without production data. All was looking great, and we launched it into production. It was feature flagged and enabled only for those in our Customer Advisory Council.

The moment we did that every single one of them had their inboxes flooded with anywhere from a few hundred to, in one case, about 15,000 emails. We’d built this notification engine to let them know when things were about to expire and embarrassingly we didn’t think about what was going to happen when things had already expired. And so every device they were managing, which counted, for many, into the thousands would suddenly trigger this alert to say, “This is about to expire”—when it actually already had. And so that was one of those cases where we went, “Hallelujah.” We had gone through a lot of exhaustive QA and unit tests, etc., so we knew things weren’t going to break and the upside was our notification engine worked at a massive scale. It still had a lot of work to do when we turned it on.

The reality was that the output for the end-user was certainly not the experience we would’ve wanted. If we had turned it on at the time to our north of 2,000 customers, we would’ve flooded our support team with a lot of requests. Also we would had a lot of very unhappy users.

“So because of feature flags we were able to quickly go back and amend that feature then test again with that Customer Advisory Council before launching as a general release.”

The real benefit obviously came from all the learning that we got from the process as well. Not just for problems, but also just day-to-day functionality that we thought we’d scoped correctly, when given to customer in production, it’s totally different.

“This is when the real learning happens. Showing them some wireframes is totally different from showing them it working with their data.”

What do you think is the number one mistake that’s made around feature flagging?

The #1 mistake for me would be implementing it and actually not remembering to ensure that the features being built are actually flagged. It sounds simple, but it’s actually a common trap that happens all too frequently. You end up not flagging enough of the functionality in the product and it becomes more work later on.

#2 You need to flag more often than you probably were planning to.

“You need to ask yourself why are these features not being flagged rather than the other way around.”

Probably the only other thing is not having a program for removing flags as well. For all the benefit it offers it does make your code base a lot more verbose, and there’s a small amount of arguable tech debt that you’re buying with all this. So not having a program for removing the flags is probably the other mistake.

How do you think feature flags play into the DevOps movement? Continuous Delivery?

Early last year I started a company called IT Glue, as the Chief Operating Officer. And I would certainly say that feature flags would be one of those reasons we were able to scale and scale so fast. We went from 500 accounts and about seven on staff, to 2,500 accounts and 63 staff within one year. So you often get asked, how were you able to scale? What’s the magic bullet?

“And I’d never say there was a silver bullet, but if I listed off the ten things that drive towards continuous delivery, a dev process supported by feature flags would always be on that list.”

They’re really a very critical part of DevOps. Having not worked in very large engineering teams I can only imagine what the impact would be and I would say it plays a fairly substantial role. But it tended to be our DevOps engineers who gained some of the best advantages from actually having these feature flags in place.

Are you seeing feature flagging evolving? If so how? And how do you expect it to change in the future?

I’ve certainly seen companies like LaunchDarkly launch into the market, which is great. So much of what was having to be done in-house is now can be managed by third parties. It’s definitely evolving, yes. I’m not seeing the evolution of it, not for the fault of anyone else, happening fast enough though. Part of the challenge, I think, is that the greater majority of companies I speak to still don’t really know and think about feature flagging as an important part of their growth strategy and engineering strategy.

Know a feature flag enthusiast we should talk to? Email us or let us know on Twitter.

17 Nov 2017

The Only Constant in Modern Infrastructure, is Change

Photo by Federico Beccari on Unsplash

We all know what an audit log is…right? We think of the audit log as a chronological list of changes in a system. Martin Fowler defines it as:

“…the simplest, yet also one of the most effective forms of tracking temporal information. The idea is that any time something significant happens you write some record indicating what happened and when it happened.”¹

While event logs will tell you that a thing happened, the audit log should also tell you what happened, when it happened, and who set that event into motion.

A Practice, Not a Product

I’m sure you can appreciate why having this level of detail is important. By understanding what, when, and who impacted an event that occurred in your system, you now have a timeline of how things have been changed, can infer why the change happened, and decide what your next steps should be.

A great example of this is with security and compliance. It’s important to show a record of what transpired and have accountability. But the audit log is a practice, not a product. You have to think about how you track and record this information so that you can access it later.

One of the things that distinguishes LaunchDarkly from homegrown feature flagging systems, is that often people who build their own don’t take the time to build an audit log. This is usually because an audit log requires a role based auth system that will allow you to track accountability. A standard database doesn’t inherently have a way to track accountability, you must add that to the schema. LaunchDarkly, on the other hand,  enforces this by requiring all users to authenticate into the system, even API access is authenticated and scopes through the use of  personal access tokens.

Let’s Play ‘What-If?’

Your team has a new feature, and you’re releasing it to your end users. You’ve decided to do a percentage rollout because you’re also testing in production—you want to be sure this new feature isn’t going to have a negative impact on any customers. You’ve also explicitly excluded some specific users. You know they’re in a quiet period, and this new feature would not be beneficial to them right now. Maybe you’ll roll this out to them in two weeks when they’re ready.

Just when you think the rollout is going smoothly, you get a call from a customer. They’ve reached out to support wanting to know what’s going on. “There’s a new feature and it’s impacting our performance!” How do you find out what happened?

What if you didn’t have an audit log? Well this process would look very different. This would be a lengthy process in which you (and your team) must look through everything to try and figure out what happened. What flag impacted the change? What was deleted or changed? Do you even have a dashboard to help you review this information? Who made the change—was it on the engineering side or the product team? How many people in support, product, and engineering do you need to talk to to sort out what could be the cause of this error? And keep in mind, you’re probably relying on people’s memory of what they might have done with 100s of flags… This is not an easy task!

The More You Know

But if you did have an audit log in place, you could quickly go back into your records and identify where a change was made, what the change was, and who made it. In this case, someone thought the rollout was doing so well, they decided to push it to 100%. However, they didn’t realize there were customers excluded from the rollout because they shouldn’t be experiencing this kind of change just yet. You could see that the exclusion rule was deleted, and you now know what you can do to fix this and move forward.

So, now I’m sure you can appreciate the value of an audit log over a simple event log. And what your team could be doing instead of hunting down elusive unknowns and what-ifs. With this valuable knowledge in hand, you can quickly identify why an event happened, and decide on how to proceed next with more confidence.

¹Martin Fowler (2004), The Audit Log

09 Nov 2017

Measure Twice, Launch Once

You want all your developers to have access to the main trunk of code to deploy — that’s the point of trunk-based development. It’s important they can put their code out as often as they want and iterate on their projects. However, you don’t always want developers turning on features that will have customer impact without some way to reverse course.

Secured activation is an under-appreciated part of feature management. Your developers can deploy code whenever they want—but when it comes time to test it externally, or turn it on for everyone, you can use settings to make sure that only a select group of people has the permissions to do so. All the activation changes should be tracked and audited to ensure that all activations have an accountability chain.

At LaunchDarkly, we have found that it’s good to be permissive about who can use and create feature flags, and restrictive about who can activate them. If you are trying to get started with transitioning to using feature flags more broadly, you might want to think about how to implement a repeatable process. You might also want to leverage LaunchDarkly’s ‘Tags’ feature to help with the organization and custom roles to assist with delegation and access.

You want the following qualities in people who have the permissions to change your user experience:

  • Understands the business reason for making the change
  • Has the technical knowledge or advisors to know when the code is ready to go live
  • Has a process in place for making the change and then testing it

You don’t want to have only one person who can do this, because they’ll inevitably become a bottleneck. Make sure your process can keep releasing even if a key team member is unavailable.

In the beginning you may look to put a process around every change and then look for optimization in that process. However, over time you should look into determining  what level of change merits process and what can be executed more easily. In some cases this might even allow for small changes to be approved or executed by individual engineers. Usually, features that have anything to do with money, user data collection, or changes in the user process should have a formal approval process. Changes to backend operations can be quieter and therefore need less formal process and lean more heavily on automated testing and peer review.

Think about your current deployment process. What happens if someone releases something too early? How do you protect against that? How will you port that control over to the access control that LaunchDarkly offers? What is the failure case if something doesn’t launch properly?

Feature flags are easy to implement in code, but managing them well across an organization takes some planning and forethought.

20 Oct 2017

Keeping Client-Side Feature Flags Secure

LaunchDarkly’s JavaScript SDK allows you to access feature flags in your client-side JavaScript. If you’re familiar with our server-side SDKs, setup for our JS SDK looks superficially similar—you can install the SDK via npm, yarn, or via CDN and be up and running with a few lines of code. However, if you dig a bit deeper into the source code and do an apples-to-apples comparison with our node SDK, you’ll see that our client-side SDK works completely differently under the hood.

There’s one simple reason for this difference—security. Because browsers are an untrusted environment, our client-side SDK uses a completely different approach that allows us to serve feature flags to your users securely. While we’ve made our approach as secure as possible out of the box, there are three best practices that we recommend all companies follow to maximize security when using client-side feature flags.

1. Choose Appropriate Use Cases

First, in a browser environment your entire JavaScript source code is fully exposed to the public. It may be minified and obfuscated, but obfuscation is not security. This means that if you’re using client-side feature flags, you must be aware that anyone can modify your source code directly and bypass your feature flags. For example, if you have a code snippet like this:

You can expect that any of your end users could inline edit the above code snippet in their browser window to show the brand new feature, bypassing the feature flag and its rollout rules entirely. This is a fact about how client-side feature flags work—not a limitation in the LaunchDarkly platform.

This doesn’t mean that client-side feature flags are always a security vulnerability—it just means that there are some use cases where a client-side feature flag alone isn’t appropriate. For example:

  • Launching a feature that is press-, publicity-, or competitor- sensitive. In this situation, having a user enable the feature early could derail a launch, or tip your company’s hand. Even the feature flag key itself (e.g. brand.new.feature) could be sensitive. In this scenario, you can’t even afford to push the new front-end code to production prior to launch, even if it’s kept dark.
  • Entitlements. A client-side feature flag alone is not a sufficient control for locking users out of functionality that they shouldn’t be able to access. In this case, a client-side feature flag is fine assuming it’s coupled to a backend feature flag. For example, you can use the same feature flag (with the same rollout rules) on both the front-end (to disable a UI element) and the back-end (to control access to a REST endpoint triggered by that same UI element).

Using client-side feature flags securely starts with choosing appropriate use cases. Ultimately, the choice depends on whether or not you can expose your new feature’s code to users that may not have the feature enabled.

2. Enable Secure Mode to Protect Customer Data

There’s a corollary to the observation that all JS code is fully exposed to the public—there is no way to securely pass authentication credentials to a 3rd party API from the front-end alone. Because anyone can view source, any API key or token that you pass to a 3rd party API via client-side JS is immediately exposed to users.

One solution to this problem is to sign requests—use a shared secret on the server-side to sign the data being passed to the 3rd party API to ensure that the data hasn’t been tampered with, or the request hasn’t been forged. Tools like Intercom take this approach. In Intercom’s case, request signing guarantees that a user can’t fetch chat conversations for a different user.

LaunchDarkly follows this approach as well. We call this Secure Mode, and when enabled, it guarantees that a user can’t impersonate another user and fetch their feature flag settings. Our server-side SDKs work in tandem with our client-side JavaScript SDK to make Secure Mode easy to implement. We recommend that all customers enable Secure Mode in their production environments.

3. Use a Package Manager to Bundle the SDK

There’s another aspect of security that comes with using any 3rd party service in your JavaScript—protecting yourself in case that service is attacked. Our recommended best practice is to embed our SDK in your code using a package manager, rather than injecting the SDK via script tag. We offer easy installation with popular package managers like npm, yarn, and bower. This does put the onus on you to upgrade when we release new versions of our SDK, but at the same time it can reduce latency to fetch the SDK and give you control in accepting updates.

Conclusion

We go to great lengths to ensure the security of our customers and their end users. We follow industry best practices in the operation of our service (we recently completed our SOC2 certification) in addition to providing advanced security controls in our product such as multi-factor authentication, scoped API access tokens, and role-based access controls. This approach to security extends to our client-side JavaScript SDK. By following the practices outlined here, you can ensure that your use of client-side feature flags keeps your data and your customers’ data safe.

18 Oct 2017

How We Beta Test at LaunchDarkly

Photo by Alex Holyoake on Unsplash

We recently looked at how some well-known companies beta test. Specifically we looked at groups that test in production, and do it well. As you know, testing in production is one of the best ways to find bugs and get solid feedback from your users. While some may shy away from this because of the risks involved, there are ways to mitigate risk and do it right. So this time we want to share how we beta at LaunchDarkly.

It’s no surprise that we dogfood at LaunchDarkly. Using feature flags within our development cycle is a straightforward process. We often push features directly into our production environment and safely test prior to allowing user access. When it’s time to beta test with users we can update the setting on the appropriate flag and get user feedback quickly. And of course, if we ever need to we can instantly turn features off.

Deciding Which Things to Build

When we’re thinking about new features to implement, we have our own ideas of which direction our product should go, but we also consider inbound requests. This can be from support tickets, questions from potential customers, or conversations with existing customers. Bottom line is we want to build a product that serves our customers, and so we do our best to listen to what they want.

Once we identify a feature we’d like to build—whether it was our own idea or a customer request—we’ll share it out to see if other customers are also interested. This is an important part of our beta testing process, because once the feature is in production and we’re ready to test it, these are the people we want to circle back with for beta testing.

Testing in Production

When it’s time to test, we test with actual end users in production. Our feature management platform allows us to turn features on for specific users. We can specify individual users, or we can expose users by attribute, like region (everyone in Denver)—and we can instantly turn them off at any time.

Because we’re testing in production, we don’t have to have an isolated environment or separate account. For those customers who showed interest, and agreed to participate in beta testing, we turn the features on in their production accounts.

Typically we beta for two weeks, sometimes as long as a month. As mentioned before, since we know which customers are interested in the feature, we can go back to them and have them test it. These are the users who already know they want this functionality, so we want to be sure it fits (or exceeds!) their expectations. And of course we want to make the most of this time, so it’s important we actually get feedback. We find that those who have asked for the feature are eager to let us know how things are working. We make a point of also following up with those who don’t proactively offer feedback—we want to hear from everyone!

While we’re testing and getting feedback, we’re taking all this information in and improving the feature before rolling it out to everyone else. When we feel confident we have something that’s ready to be shared, we’ll begin a percentage rollout to the rest of our users.

Embracing Failure

Using feature flags around features within our development cycles allows us to mitigate risk by pushing out small, incremental changes at a time. As you can see, this also enables us to beta test quickly and safely. If there are major bugs, we’re more likely to identify them early on before affecting our all of our customers.

“Embrace failure. Chaos and failure are your friends. The issue is not if you will fail, it is when you will fail, and whether you will notice.” -Charity Majors

Right now we’re currently in beta for scoped access tokens and a new faster .net SDK. Let us know if you’d like to take a look at it early, we’d love to hear what you think.

02 Oct 2017

Removing Risk from Product Launches: A webinar with LaunchDarkly, CircleCI, and GoPro

We recently sat down with one of GoPro’s Senior Engineering Managers, Andrew Maxwell, and the CTO of CircleCI to discuss reducing risk in product launches. Andrew talked about how his team delivered their code two weeks early, tested in production, and had an overall successful product launch. He goes into detail, sharing how his team uses continuous integration and feature flags to make product launches like that possible.

The Big Launch

Andrew’s team is responsible for web applications. Last September his team was focused on a big initiative around a product launch called GoPro Plus, which allows users to access and share content wherever they are. This launch included both mobile and desktop apps, and promoted two new cameras in the GoPro line.  

Two Weeks

The team delivered their code and pushed it into production two weeks early:

“…we used LaunchDarkly to push our code to production, turn the apps off — off by default—and then make sure that we had everything pushed out, deployed, and the infrastructure running live without customers actually seeing it.”

Two weeks gave them time to do a full integration test their features. They tested both in-house and in production—slowly opening up who had access—so they could get valuable feedback, find bugs, and make continuous improvements in the weeks leading up to the big launch. On the day-of, they were confident in their work and simply turned on 12 feature flags.

Check out the full webinar below to learn more about how the team used CircleCI and LaunchDarkly, their planning strategies, and best practices for their continuous integration pipeline.