16 Feb

Net Promoter Score + Feature Flags for canary releases

Canary releases are a DevOps best practice of pushing new features to a subset of your customers. By pushing to a subset, this group can provide early feedback, verify functionality, and act as “canaries” to your entire population. Originally, canary had a negative connotation – miners took canaries down into mines, as canaries were more sensitive to bad air. If the canary stopped singing, the miners knew it was time to skedaddle.

Surprisingly, being a canary can actually make your customers even bigger fans! I heard a great use case from a LaunchDarkly customer People.ai,  AI for managing sales teams, who would automatically push new features out to users who had an NPS of over a certain threshold. Net Promoter Score (NPS) rates how strongly would a customer recommend your solution, so initially, this seemed bizarre.  Why risk showing your best advocates functionality that might be unstable, early, or fragile? However, the people who are happiest users (likely to recommend) are actually very excited about being the first to see new functionality. They love feeling plugged-in to new development. And in addition, with LaunchDarkly feature flag management, if a feature isn’t doing well, the team can instantly turn it off, reverting to the old, stable experience.

I love hearing stories about how customers are using LaunchDarkly’s segmentation capabilities to release better software, quicker.


14 Feb

Implementing feature flags in single page apps, using React and Redux Saga

Built on frameworks like React, Angular, and Ember, single page apps (SPA) are becoming the new norm for modern applications. They center around dynamically updating a single HTML page via AJAX and HTML5 to deliver a more fluid and faster user experience. This introduces some new complexity when it comes to controlling access to front-end features, specifically via feature flags.

Feature flags (toggles) are used to gate access to particular code snippets, allowing you to control a feature’s rollout, target specific users, and kill a feature in production. The challenge with feature flagging single page apps is handling the state transformations (the changes in a webpage’s DOM) in a way that maintains performance and a fluid user experience.

This article discusses how to feature flag in a React single page app to show best practices. More specifically, we demonstrate feature flagging using LaunchDarkly’s JavaScript SDK.

Feature Flagging in Single Page Apps

With the help of React Router, creating a single page application can be a very quick and easy process, allowing developers to minimize risk and test features without degrading the user experience. We use Redux Sagas to handle asynchronous actions, which is perfect for integrating our app with the LaunchDarkly JavaScript SDK.

Feature Flags

In our app, we use two feature flags, user-type and header-bar-color. The user-type feature flag controls the content that is displaying depending on the group of current logged in user, while header-bar-color returns a hex code to toggle the color of the navigation bar.

An implementation of header-bar-color might look something like this:

Integrating Feature Flags with Redux Sagas

We handle the asynchronous process of requesting a LaunchDarkly feature flag by using Redux Sagas. The workflow begins in the App component of our SPA, where we dispatch an action to initialize the LaunchDarkly client.

Then, we can use a saga as middleware to wait for an initialization request to be received, using redux-saga’s takeEvery module, which will call an effect, initLD.

The effect will call getLD, a function which creates a promise, waiting for the LaunchDarkly client to be ready for flag requests. Since the user is anonymous, we can generate a random token to be used as their key.

Once the client is ready, the effect will send the action off to our reducer, to store the client and current feature flag state in props.

Now that the LD client is initialized, and stored in our props, we can define a process to update our flag whenever the current user changes (logging in or out, for example). We take advantage of the fact that we’re already using sagas to asynchronously handle user authentication. When logging in, the authorize effect is called to ensure the login data is correct. When authorization is complete, we’ll make a call to idLD. This will identify a new user in our LaunchDarkly client, and send the correct flag to the reducer for our newly logged in user. Logging out works in a similar fashion. (Note: We make use of some utility functions defined in auth, which return user information from our local storage.)

So, what do we do with the flags stored in our props? Well, in our Home component, we’ll render a different page depending on the flag that was received. In our case, if a fallback flag is received, the “Anonymous” page will be shown no matter who is logged in.

Meanwhile, every time a page is loaded, the Nav component takes in the headerColor prop and changes the background of our navbar to said color.

See it in action

What’s next?
There are many ways to feature flag within a single page app, depending on your case and the complexity of your features. You can also check out this repo by TrueCar, which provides another avenue for feature flagging in React Redux.

Want to try out this feature flagging in React for yourself? Check out the open source repo!


Primary Contributor:  Arnold Trakhtenberg, Engineering Content Consultant at LaunchDarkly

10 Feb

LaunchDarkly Values

LaunchDarkly Team
  • Respect and integrity for our team, our customers, and our community.
  • We believe in teams, not fiefdoms. Leaders, not tyrants.
  • We’re building a place where you can learn and grow.
  • Work is not life.

When John & I started LaunchDarkly, we’d known each other for years, since being undergraduates at Harvey Mudd College. Part of why we started the company together is that we shared common values about the company that we wanted to work at. As a founder, you’re not only building a product – you’re building a culture.

As we got bigger, it was time to formalize our values from tribal “this is the way we work” knowledge to something more scalable. We surveyed our current team, and I was thrilled that, what I thought our values should be, the team already thought were in place. We reviewed our values at our last team meeting, and everyone gave an example of the time they’ve seen them practiced.

“Values are valuable when you have a hard decision to make,” is John’s and my view. By making our values public, we hope both our customers and our team can know who we are.

08 Feb

And so the adventure begins

It’s something of a LaunchDarkly tradition to write a blog post at the end of your first week. But here I sit, deep into week number two, and I haven’t written a thing. In my defense things move pretty fast around here, and I was just trying to keep my head above water. My second day on the job, having finally set everything up so I could start coding, they had me deploy production. By the time Friday rolled around, I had fixed 3 bugs, opened 6 pull requests, deployed production two more times — and written zero blog posts. Better late than never though, right?

A new chapter

I’m extremely thrilled to be part of this great company. I joined LaunchDarkly after nearly 10 years at Atlassian, where I spent most of my time doing front-end development and arguing about UX. I’m excited to take everything I learned there and continue to make LaunchDarkly a product people love to use.

I took the job because I believed in the people, the product, and the culture. If anything I underestimated all of those things and I can’t wait to see where this journey takes me.

An escape

The other day we went on a team outing to an escape room. If you’re not familiar, you basically solve a series of puzzles using items in a room in order to “escape” the room. I’ve never done one before, so I have nothing to compare it to, but this particular escape room required a lot of coordinated effort. We had 12 people participating (almost the whole team, which is at 15 now), but at any given time there were probably 3-5 puzzles being worked on.

The beginning was a little chaotic as everyone tried to figure out exactly what we’d gotten ourselves into and what in the world to do next. But pretty soon we settled down, started self-organizing, and talking through the problems. It was pretty amazing to see everyone working together so well — no arguments or drama, no jockeying for leadership, just good old-fashioned teamwork. I think that experience sums up my time at LaunchDarkly so far — fast and fun, with a great group of people working together towards a common goal.

Oh yeah, and we escaped the room (just barely).

08 Feb

Toggle Talk with Slack’s Director of Engineering Josh Wills

I sat down with Josh Wills, Director of Engineering at Slack and unabashed feature flag enthusiast, to get his opinions on the practice, where it’s most useful, and how it has played an important part in his career.  I think my favorite take-away from him was,

“Feature flagging is scary. I get why it’s scary. But to me, not launching your product every five minutes is scary… Launching continuously is how you learn fast– It’s not just about deploying fast, it’s about learning fast. That’s the future of viability.”

Here’s what I heard from Josh in the interview.  


How long have you been feature flagging?

I started feature flagging at Google in 2007, and when I joined the team they were in the middle of rewriting the feature flag system. Our feature flag framework was called the Experiments Framework because it was designed for running A/B tests, and it grew out of that into this very powerful and complicated feature flagging framework that we used for literally everything. It started out as a simple “feature on”/“feature off” system, but it evolved to include richer types like strings and floats and even had some simple conditional logic for modifying flags based on attributes of the request. It became the system through which all of Google’s various machine learning models were combined to make decisions for ad ranking, search ranking, etc.

Imagine that you have dozens of machine learning models active on a given request, doing all kinds of different things, and your job is to figure out an algorithm for deciding which ads should go where and how much each advertiser needs to pay. All of those different signals are combined together through a complicated series of equations which have a bunch of thresholds and weights. There’s very rich logic in not only the machine learning models, obviously, but also within the feature flag framework, to control under different contexts what counts the most.

Trying to do data science and machine learning in production without feature flags is nuts.

The feature flag framework here at Slack was developed in-house and was not initially developed with data science in mind, but we eventually created one that was to support our own search ranking, backend performance, and new team onboarding experiments.

But when I got here, I was so happy that they had a feature flag framework at all– it means you deploy code hundreds of times a day, not once a month, or whatever it is other companies do.


What do you prefer to call it and why?

I like to call it the experiment framework, but that’s just Google/Xoogler nomenclature. Facebook’s system is called Gatekeeper and it’s basically the same idea. Most of these systems have converged to have the same set of features, because it just makes sense and you have to have certain things. Eventually you’ll get there anyway, so why not just skip to the end?  


When do you think feature flagging is most useful?

I’m biased, since I’m a data person. Data science, machine learning is what I do and I think it is absolutely critical for machine learning, for bringing any kind of data driven feedback loops and intelligence into your application. That is when it is absolutely most critical.

You should not be doing machine learning without a feature flag framework.

If you are saying, “Oh I want to get into machine learning, and I’m going to do predictions or personalization or recommendations,” which lots of people do, and you are doing it without a feature flag framework, you are insane and should be fired. Not to put too fine a point on it, but…


Are there any cases where feature flagging is not a good idea?

Well, there is always tech debt that goes along with doing this kind of stuff. We deal with this at Slack, and Google deals with it as well. You end up with code that has a lot of “if” statements in it. Engineers are not always the best about deleting their feature flags once they’re no longer necessary.

So we have a whole archival system, so that when you do the code review to create the feature flag, you also have to specify when you plan to delete the flag, and get alerts if the flag is not removed by such-and-such a date. That is the cost of doing business. I would say the benefits massively outweigh the costs.

I think there are certain other situations where feature flags are not a good idea.  They’re relatively rare, but they do happen. Where you’re switching backend systems there can be times where you have to just go for it. You maybe reach a point where you just can’t quite bring yourself to burn the bridge and just go without the old system anymore, even though it’s causing you pain, and you know it needs to go. You just need to bite the bullet and do it, do the cut over and live with the consequences.

You work super hard to not ever find yourself in that situation. No team is going to go out of their way to corner themselves like that, but if you’re cornered, you’ve got to fight your way out.


Best use of a feature flag – a personal story?

It’s been crucial to the way I’ve worked ever since I moved to San Francisco. Coming from the perspective of doing machine learning, data science, etc. at Google, I ran thousands of experiments all in search of new ways of combining different models together, in order to fundamentally make Google more money, make the user experience better, and generate more ROI for advertisers. I got really good at it. You could say that feature flags made my career.

At the same time, there was one Friday afternoon back in 2009 where I thought it would be a good idea to do one last push, and I launched a bad feature flag configuration that broke the ads systems and cost Google a lot of money. I remember my boss saying at the time, “So Josh, what did you learn from the X million dollars we just spent educating you?”

It’s the kind of thing where you learn that canaries are good, and that 5 pm pushes on a Friday are pretty much always a bad idea, no matter how good your feature flagging framework is.


What do you think is the number one mistake that’s made around feature flagging?

I think my biggest thing is if you are really thinking of feature flags as just “on” “off” switches, you are missing the real power of what they can do.

To me the feature flag space is the parameter space that I get to explore to optimize whatever metrics that I want to optimize. If you are constraining yourself to this very limited boolean on/off space without strings, floats, etc. you’re putting artificial limits on how fast you can explore the space and about how all of the knobs at your disposal work.

This will be the early mistake that I think a lot of people make– they won’t feature flag often enough.


Can you share any tips for better flagging?

I think what was most compelling for me at Google was the configuration language for feature flags was very rich, but not Turing complete. I don’t think that configuration-as-code is a good idea for feature flags, because it becomes harder to test/validate, which slows down how quickly you can roll things out. However, the configuration language was like programming in the sense that you could define what we called “condition functions” that could be evaluated in the context of a request and used to adjust the values of the flags. So the logic was like a long series of “if” statements, where you could modify the resulting value either by overriding it or by addition, multiplication, or other custom operators.

The way it worked when I was at Google and working on the ad system was that the server binary was pushed weekly, so the only changes that you could make during the week were via experiments. Having that sort of richness of programming via feature flags allowed a lot more freedom and a lot rapid safer iteration between those weekly pushes. The same logic applies today for mobile apps, where you can only do releases via an app store, but you want to be learning what works much faster than that.


How do you think feature flags play into the DevOps movement? Continuous Delivery?

How would you do anything without them?

What does it mean to do DevOps without feature flags? It’s one of those things that doesn’t make sense to me. How would you make mayonnaise without eggs? It’s like, not really mayonnaise then. Which is good, because mayonnaise is gross.

I would be fascinated and somewhat horrified to have someone try to explain a feature flag-less DevOps set up. I don’t know what would that look like.


Are you seeing feature flagging evolving? If so how? And how do you expect it to change in the future?

Not as much as I would like, broadly speaking. I’m not at Google anymore, so I don’t know what the current state was when I left. The stuff they had is much richer than anything I’ve seen anywhere else, but to be fair, they were doing a lot of machine learning much earlier than anyone else, too.

I think feature flags need to find a way to strike the balance between configuration as on/off switches and configuration as Turing-complete programming language. That was the thing I felt was most compelling and powerful about the way Google did it that I’ve not seen anywhere else.


03 Feb

My First Week

Start your Engines

As I reminisce on my first week at LaunchDarkly, I’m reminded about the initial call I had with Russ, our VP of Sales, to learn more about the company and opportunity. Russ and I have known each other since 2012, when I interviewed for his team at Box. Although we didn’t end up working together, we kept in touch and kept joking that one day, we would! I guess you could say it was fate that caused that call a few months ago to happen…or just perfect timing.

When I spoke to Russ and learned more about LaunchDarkly, I immediately felt that gut instinct say “yes”. It was clear to me that this represented a great opportunity to join an awesome team that had built an amazing product that customers loved. Now, I look back on my first week as validation for the gut feeling I first had.

Shake and Bake

The whole week was filled with highlights. Monday started out with our first revenue meeting, then an all hands where John and Edith walked us through the company values (which I agreed with wholeheartedly). I sat in on a variety of customer calls, which showed me how interested companies are to speak with us and how valuable our platform is to them. The sales team met with one of our advisors (a former co-founder / CTO) and had a great meeting learning about how best to partner with VPs of Engineering / CTOs. And then on top of that, I learned more about the product, the market opportunity, and the company vision.

What I’m left with after that first week is being continually impressed by the company and product John, Edith, and the team have built. From hearing successful customer stories, meeting other team members, and learning about the strength of the product, I’m filled with confidence in our ability to succeed. More so, I’m excited to start partnering with companies and help them deliver better software, faster.  It’s been a great first week, and I’m looking forward to many more.