02 Jun 2017

Week n+1

Start? This is just the next step in the journey (image credit: Andrew Lipson)

I recently wrapped up my first official full week at LaunchDarkly. Although, I’ve been working with the team as an advisor/consultant for a number of months. Over the past year I have been advising and consulting with various start-ups looking for a good fit. I would often remark to folks that I was “company dating.”

More like introductions from friends than tinder. (Photo credit: Reddit post)

Honestly, I have very little dating experience. My wife and I met my 2nd (her 1st) year in college and haven’t looked back. Similar on the job side, I started at EMC a year after graduating and then was the first internal transfer to VMware. For almost 15 years I enjoyed the stability and resources of a large company. But, then I started to feel the need to grow and participate in the changing landscape I saw in software development trends.

Since leaving VMware I have spent a lot of time thinking about the gap between the old school development frameworks (e.g. waterfall) and newer practices (e.g. agile, scrum, continuous deployment). Tools like git, continuous integration, and automation have radically changed how we release. At EMC and VMware we measured our releases in years, or sometimes months (the same way a parent refers to their 22 month-old toddler). Compared to GitHub where we released multiple times a day.

This whole cloud thing is likely just a fad. (Photo credit: Twitter)

Recently, I’ve started to bucket these tools into three phases of for software development: Concept, Launch, and Control. I’m working on a blog series to discuss each of these in-depth, but this framework is what got me excited about LaunchDarkly. Feature management, while not the shiniest tool, provides the foundation for eliminating risk and delivering value as teams push to move faster and to be more reliable.

In addition to my passion for our product, this intelligent team, my carless commute, I did have one additional objective: to be a part of a diverse and equitable company. Not simply an organization that accepts diversity, but one that actively pursues a more diverse and inclusive team as a imperative for building better products and services. So far a great start to my next long-term relationship.

12 Apr 2017

How Spinnaker and Feature Flags Together Power DevOps

It’s very common for customers to be excited about both Spinnaker (continuous delivery platform) as well as feature flags. But wait? Aren’t they both continuous delivery platforms? Yes, they are both trying to solve the same pain points – the ability to quickly get code in a repeatable, non-breaking fashion, from the hands of the developers into the arms of hopefully excited end users, with a minimal amount of pain and heartache for everyone along the toolchain. But they solve different pain points:

  • Spinnaker helps you deploy functionality to clusters of machines.
  • Feature Flags help you connect those functionality to clusters of USERS.  

Spinnaker helps with “cluster management and deployment management”. With Spinnaker, it is possible to push out code changes rapidly, sometimes hundreds (if not thousands) of times a day. As Keanu Reeves would say “Whoa.” That’s great! All code is live in production! Spinnaker even has handy tools to run black/red deployments where traffic can be shunted from cluster to cluster based on benchmarks. Dude! For those who remember the “Release to Manufacturing” days where binaries had to be put on an FTP server (and hope that someone would install and download in the next quarter or so), code being live within a few minutes of being written is amazing. For those who remember “master disks” and packaged software, this is even more amazing.

Nevertheless, with dazzling speed comes another set of problems. All code can be pushed anytime. However, many times you do not want everyone to have access to the code – you want to run a canary release on actual users, not just machines. You might want QA to try your code in production, instead of a test server with partial data. If you’re a SaaS product, you might want your best customers to get access first to get their feedback. For call center software, you want to have an opportunity to test in a few call centers. You might want to have a marketing push in a certain country days (or weeks or months) after another country. You might want to fine-tune the feature with some power users, or see how new users react to a complicated use case. All of these scenarios can not be done at a server level. This is where feature flags come in. By feature flagging, you can gate off a code path, deploy using Spinnaker, and then use a feature flag to control actual access.

Together, Spinnaker and feature flagging make an amazing combination. You can quickly get code to “production”, and from there decide who gets it, when.

14 Feb 2017

Implementing feature flags in single page apps, using React and Redux Saga

Built on frameworks like React, Angular, and Ember, single page apps (SPA) are becoming the new norm for modern applications. They center around dynamically updating a single HTML page via AJAX and HTML5 to deliver a more fluid and faster user experience. This introduces some new complexity when it comes to controlling access to front-end features, specifically via feature flags.

Feature flags (toggles) are used to gate access to particular code snippets, allowing you to control a feature’s rollout, target specific users, and kill a feature in production. The challenge with feature flagging single page apps is handling the state transformations (the changes in a webpage’s DOM) in a way that maintains performance and a fluid user experience.

This article discusses how to feature flag in a React single page app to show best practices. More specifically, we demonstrate feature flagging using LaunchDarkly’s JavaScript SDK.

Feature Flagging in Single Page Apps

With the help of React Router, creating a single page application can be a very quick and easy process, allowing developers to minimize risk and test features without degrading the user experience. We use Redux Sagas to handle asynchronous actions, which is perfect for integrating our app with the LaunchDarkly JavaScript SDK.

Feature Flags

In our app, we use two feature flags, user-type and header-bar-color. The user-type feature flag controls the content that is displaying depending on the group of current logged in user, while header-bar-color returns a hex code to toggle the color of the navigation bar.

An implementation of header-bar-color might look something like this:

Integrating Feature Flags with Redux Sagas

We handle the asynchronous process of requesting a LaunchDarkly feature flag by using Redux Sagas. The workflow begins in the App component of our SPA, where we dispatch an action to initialize the LaunchDarkly client.

Then, we can use a saga as middleware to wait for an initialization request to be received, using redux-saga’s takeEvery module, which will call an effect, initLD.

The effect will call getLD, a function which creates a promise, waiting for the LaunchDarkly client to be ready for flag requests. Since the user is anonymous, we can generate a random token to be used as their key.

Once the client is ready, the effect will send the action off to our reducer, to store the client and current feature flag state in props.

Now that the LD client is initialized, and stored in our props, we can define a process to update our flag whenever the current user changes (logging in or out, for example). We take advantage of the fact that we’re already using sagas to asynchronously handle user authentication. When logging in, the authorize effect is called to ensure the login data is correct. When authorization is complete, we’ll make a call to idLD. This will identify a new user in our LaunchDarkly client, and send the correct flag to the reducer for our newly logged in user. Logging out works in a similar fashion. (Note: We make use of some utility functions defined in auth, which return user information from our local storage.)

So, what do we do with the flags stored in our props? Well, in our Home component, we’ll render a different page depending on the flag that was received. In our case, if a fallback flag is received, the “Anonymous” page will be shown no matter who is logged in.

Meanwhile, every time a page is loaded, the Nav component takes in the headerColor prop and changes the background of our navbar to said color.

See it in action

What’s next?
There are many ways to feature flag within a single page app, depending on your case and the complexity of your features. You can also check out this repo by TrueCar, which provides another avenue for feature flagging in React Redux.

Want to try out this feature flagging in React for yourself? Check out the open source repo!

—-

Primary Contributor:  Arnold Trakhtenberg, Engineering Content Consultant at LaunchDarkly

23 Jan 2017

Microsoft VSTS and LaunchDarkly Team Up to Control Releases

LaunchDarkly and Microsoft controlled rollouts and use targeting feature toggles and feature flags

It’s every project manager’s dream to get complete control over their software releases and have a way to continuously integrate and deliver new features, while also controlling the visibility of those features once they’re live.

Microsoft Visual Studio Team Services

Microsoft’s Visual Studio Team Services (VSTS) enables development teams to build, integrate, test, and release new software in a single platform. It has a centralized version control system that allows for continuous integration, package management, and release management. Together, these components enable agile teams to move faster in a more centralized manner while also minimizing the number of third party dependencies needed for deployment.

Traditionally, when you release new features into your Production environment, they are live for all of your users. You could provide code-level access control via a config file or modify database values to superficially control the visibility of your new features. However, these release controls are engineer-dependent, meaning they require a developer to make changes to who gets to see a feature and when. It makes it very difficult to target granular segments of users or perform incremental percentage rollouts to test performance and visibility.

LaunchDarkly VSTS Extension

With LaunchDarkly, you can use feature flags to control the full release lifecycles of your features, perform percentage rollouts, and granularly target user segments. These controls are surfaced via a UI that can be used by non-developers, including designers, project managers, and the business team.

LaunchDarkly and Microsoft Visual Studio Team Services (VSTS) Feature Flags / Toggles for release management

To tie this feature flag UI into your VSTS workflow, the LaunchDarkly Extension  allows you to associate feature flag rollouts with VSTS work items to get complete control over who sees what, and when. This allows teams to:

  • Centralize feature and release lifecycle management
  • Release confidently with less risk
  • Incorporate feature flags into the release process
  • Team support for superior project management
  • Gain better visibility into features and outcomes
  • Achieve next-level continuous delivery
  • Empower your team to do better DevOps

Here’s some of the core functionality when you integrate VSTS and LaunchDarkly:

Associate feature flags with work items

Take full control of the release lifecycle of your work items and manage feature rollout

LaunchDarkly and Microsoft Visual Studio Team Services (VSTS) Feature Flags / Toggles for release management work items

Comprehensive Release Management

Manage percentage rollouts and turn the feature flag on or off

LaunchDarkly and Microsoft Visual Studio Team Services (VSTS) Feature Flags / Toggles for release management controlled percentage rollouts

 

Incorporate feature flags into your release definitions

Tie feature flags to your Visual Studio Team Services release definitions and perform percentage rollouts:

LaunchDarkly and Microsoft Visual Studio Team Services (VSTS) Feature Flags / Toggles for release management work items and rollouts

Getting Started

If you already use VSTS and have a LaunchDarkly account, then all you have to do is connect the LaunchDarkly extension and you’re ready to feature flag and take total control over your releases.

You can also check out the documentation to learn more about setup and integration.

05 Dec 2016

Cultural Changes of Feature Flagging vs Branching: Defrag X

I was honored to give a talk at Defrag X on “The cultural changes of feature flags vs feature branching”. Feature flags initially enabled developers to separate deployment from release. Now an entire organization (product, marketing, sales, customer success) can shift from a waterfall way of thinking to a more iterative, customer- centric release. Afterwards, I was thrilled by how many people came up and said they appreciated my talk. 

When I first stDenverDefragarted LaunchDarkly, I met with an Engineering Director, who was excited about feature flags but was still on a six month release cycle. That same Engineering Director was at Defrag. He’s now at a different company with a quicker release cycle and is ready to move forward with feature flags. I’m happy that as release cycles get quicker and quicker, so to does the need for feature flag management.

Thanks to Defrag for ten years of great conferences, with even a bonus snow appearance.

29 Nov 2016

Toggle Talk with Damian Brady

I sat down with Damian Brady, Solution Architect at Octopus Deploy for a conversation about his experience with feature toggles.  He shared with me his tips for best practices, philosophies on when to flag and what he thinks the future of feature flagging will look like. 

  • How long have you been feature flagging?

I had to think about this one a bit – about 8 years ago but I probably didn’t know what it was called at the time.

“It’s definitely the case that people are doing this without knowing the name “feature flag” or even giving it a name. They’re just saying it’s a configuration switch or a toggle and but not giving it a more proper name, they’re not identifying it as a first-class citizen really.”   

  • What do you prefer to call it and why?

Now I call it feature flagging or occasionally feature toggles. I think toggles makes a bit more sense as analogy for non-technical people.  

  • When do you think feature flagging is most useful?

There’s a couple – but the one I think it’s most useful for is to use a feature flag when you have a feature that is nearly complete or complete from your point of view. Either way, you are ready to get verification from someone with real data.

“You can test as much as you want with your pretend fake data, or even a dump from production which is being obfuscated, but until it gets used in the wild you’re never really sure that the feature is doing exactly what it needs to do.”  

So hiding that behind a feature flag, and then clicking it on for somebody who is using the product for real in any way gives you that last little test that is ultimately the most useful.  At that point you still have the opportunity to back out. If something was corrupt or your expectations were wrong, it’s really useful for that last-minute check.  

At Octopus, we’ve started using feature flags for big features that a lot of people don’t want to see. So a while ago we introduced the idea of a multi-tenant deployment. And probably most of our users don’t need that feature because it adds a lot of complexity to the UI.  We have a configuration section where you can toggle an “on” and “off” switch, so if you don’t need that feature you can just leave it off.     

Are there any cases where feature flagging is not a good idea?

I think there are two extremes where feature flagging is not a good idea. On one hand, flagging really small changes can be more trouble than it’s worth. It’s introducing an extra level of complexity that maybe for a small change is not critical.  

On the other side, using feature toggles around the architectural changes in the core of your application – that’s kind of hard to test. Do you have a feature flag that when you turn it “on” it completely redirects the way the entire application will run? In that case you bite the bullet and decide that this is a big change and you’re just going to have to test it very thoroughly and not give yourself a way out.

That being said, there are some cases where you still need to give yourself a way out by using a flag. For example, you might deploy some new feature thinking it’s correct, but subsequently learn from a customer or user that it doesn’t really meet their needs. Rather than the user living with a bad feature, you might want to turn the flag off and go back to the drawing board.

If it’s an architectural change, you may only find out that there’s a bug when you use it in production. Test data may not surface the issue properly.

Ultimately, doing core architecture changes in a way where you can back out later can be an extra huge amount of work. It’s probably at that point you know you aren’t going to do it (revert back) anyway.   

  • Best use of a feature flag – a personal story?

When I first started using feature flags, around the 8 years ago timeframe, I was working on a web application that was internal and a big line of business.  And we had just added a new third party provider for providing SMS.  And with this new provider, it meant we had to write a lot of new code.  It was internet banking software so it was a one-time password we were sending out – and it was really really important that it work.

We tested everything rigorously but wanted more insurance.  So we put the new service behind a feature flag. We had a bunch of agents that ran this type of SMS. We enabled a flag for one of the agents and monitored it to make sure it was actually doing the right thing and not failing. And then we started trying other ones. It failed a couple of times because of differences in the sandbox environment between the third-party provider and the real one.

“We thought everything was okay, but when we put it live we turned it on slowly, and it didn’t do what we expected.”

So when that happened we turned it back off again…and went back to the drawing board.

So without the feature flag, we would have dropped every person using the service at that critical point. That client would have not been able to receive SMS’s until we were able to rollback.

  • What do you think is the number one mistake that’s made around feature flagging?

There is one that I keep seeing – when you wrap a new feature you believe to be finished in a flag, the biggest mistake with this is not testing that change with the flag “on” and then “off”. For instance, when you turn it “on” it snaps into new database tables or starts changing the way data is saved. But when you turn it “off” again, you’ve lost that data or data is corrupt. For this you need to test it “on” and “off”.  

“If you have more than one feature flag running at the same time, test the combinations of them being both ‘on’ and ‘off’.

If they’re likely to interact with each other you need to test “one on, one off,” “both on,” “both off” and all possible combinations like this.  

  • How do you think feature flags play into the DevOps movement? What about Continuous Delivery?

I think feature flags play in both continuous delivery and continuous deployment. I think they’re most useful to continuous deployment. You have all of your features pushed out to production as soon as they compile essentially – but they are behind a feature flag so you don’t break anything. That’s the way Facebook does it. They know that any new code they write might end up in production so it’s going to be safe behind a feature flag.  

“The design of the DevOps movement, the aim of it really, is to get real features and real value in the hands of users as quickly as possible.”

So if you have to wait until this “half done piece of work” is actually safe to deploy then that slows you down. So having it behind a feature flag so that it doesn’t get touched until you are ready to test it can be really powerful for increasing velocity and getting things out to production much faster.     

So even for marketing teams, it means they don’t have to tell the developers “hey we worked out the result of this a/b test and we want option b.” If the marketing department can just just flip that switch and say “no, option b is working better so just leave it there” without a new deployment or contacting the developers to remove the old stuff and redirect to the new stuff,  that increases that team effort of getting value to customers which is the whole purpose of DevOps.

  • Can you share any tips for better flagging?

If you’re feature flagging a big change, pair the feature flag with a branch by abstraction pattern. See the clip from my talk from NDC Sydney for more details.

There’s also the concept of transitional deployments – again refer to my video clip here for more. It’s useful for things like database schema changes where you have a midpoint for both the new and old applications that will work with the schema that’s currently there. So you can turn that feature off if you need to.

  • Are you seeing feature flagging evolving? If so how?  And how do you expect it to change in the future?

It’s been around for a long time…but I think it’s becoming much more visible – and partly it’s LaunchDarkly helping with that. I think more people will start using feature flags in their continuous delivery pipeline. And the more continuous delivery becomes mainstream, the more mainstream developers will need feature flags.  

“I think feature flagging is starting to be something that you have to add your deployment cycle because you know it needs to be fast and you know you feature needs to get to production as quickly as possible – and feature flags are the way to do that.”  

So as it becomes more mainstream I think there will be more tools, more frameworks, more awareness of it (feature flags) as it hits more and more companies. I think there will be things coming out like feature flag-aware testing tools – so testing tools that know that they need to test with this flag on and off.  

The summary – more tools around best practices around this thing which is becoming more mainstream.  With DevOps becoming more popular, more people are thinking “yes we need to get to production quicker, we need that cycle time to reduce” so it’s a natural extension I think to start solving some of those problems with feature flags.  

“I think it’s just starting to become more mainstream frankly because it’s a solution to a problem that is starting to become more mainstream.”