04 Dec 2017

LaunchDarkly, #1 Feature Management Platform, Gets $21M in Series B Funding

As founder and CEO of the leading feature management platform, I’ve seen how our customers use LaunchDarkly to help them innovate more quickly, reduce risk, and break down the barriers between developer, product, marketing, and sales. Ultimately, LaunchDarkly helps our customers around the world help their own customers succeed. We take the feature flagging platform that the biggest tech giants (Facebook, Amazon, Netflix, Google) build in-house, and provide it as a service for everyone else. Thanks to a tremendous response—we serve over 10 Billion (with a B) features EVERY DAY— I’m pleased to announce that we’ve raised $21M in our Series B to accelerate our own growth in engineering, customer success, and category education.

Feature flagging/toggling is a deceptively simple idea—by separating code deployment from release with a “flag” or ”toggle”, companies can control who gets what feature in their software. LaunchDarkly allows customers to manage their feature flags at scale, giving them the in-house platform that the big tech giants have. Companies start by using LaunchDarkly for an initial “dark launch” by selectively releasing a feature to a group of their customers. This is something tech giants like Facebook and Netflix do constantly, and it allows them to manage what features we see and use in their products with minimal risk to their business. Once they become comfortable with our platform and services, the product team is able to use LaunchDarkly feature flags for fast feedback, marketing team can use LaunchDarkly for betas and launches, and sales can use us for contract management.  

LaunchDarkly is a unified platform where developer, product, marketing, sales, and customer success teams can manage code in real time. Our three main types of customers are:

Disruptors
These startups want the same feature management superpowers Facebook and Netflix have. We’ve worked with startups as they’ve grown from four people to thriving successful businesses, like Troops.AI. They’ve used us for every feature, usually starting with risk mitigation, then moving into limited rollouts, and then allowing everyone in the business to control their own features. As one startup company’s CEO told me, “We originally started using you as an “oh shoot” button, now we use you everywhere.” Another VP Product said, “Using LaunchDarkly for feature development is like night and day.”

Transitioners
These customers built their own feature management infrastructure and are tired of maintaining it. When companies like Lanetix and a leading ecommerce car buying portal made the switch to our platform, suddenly their developers could do all the things they wanted, like role based authorization and complex rule sets. And what’s more, the rest of the company can also use our tools. Now developers can focus on building and the entire company benefits from access to control and flexibility. When I was in Australia, the company told me “now when we build a feature, everyone asks ‘did you LD it?’”

Innovators
These modernizing companies know they need to move faster to innovate. They are at the forefront of their industry and know that constantly iterating will help them stay competitive. Last year, I gave a talk at NDC Sydney on how to use feature flags. An engineer from a huge IOT conglomerate immediately asked me for a demo and became a customer. This year, he gave a talk on how he’d moved from annual releases to weekly releases.

It’s extremely gratifying for our entire team at LaunchDarkly to see how much customers rely on us to run their own businesses. Customers small and large are looking to us not just as a developer tool, but as a platform that their entire company can use to deliver functionality to the right person at the right time.

While we were in Sydney, a customer’s CEO sent me a personal thank you note for a sales person visiting them and educating them on how best to use feature flags. If you’ve been in enterprise developer software, like I have, you know that usually the reaction to a salesperson visit is not kudos. However, our customers view us as a trusted advisor for expertise in feature management. I am so proud of our team and I hope our funding will help us continue to be the #1 trusted feature flag management platform, as well as invest in more education for our customers and broader market. Want to join our team? We’re hiring!

We found perfect partners in Scott Raney (Redpoint) and Jonathan Heiliger (Vertex). Scott has been a long time friend of LaunchDarkly, giving me advice and guesting on my podcast, “To Be Continuous”. Jonathan is a cloud infrastructure pioneer who is very familiar with the value LaunchDarkly provides from his time at Facebook. I’m looking forward to working closely with them both through the next chapter of LaunchDarkly.

So what’s next? LaunchDarkly has an incredibly broad base of cross-industry customers, from banking to insurance to shipping to ecommerce to hardware. The appeal of feature management is truly game-changing. Instead of code being a static object that’s changed only once a year or quarter, suddenly, code is a living, evolving power. Developers can build, marketing can launch, product can iterate, and sales can sell. Equipping businesses with the ability to move at the speed of every deploy allows an entire company to learn rapidly, deliver value to their customers faster, and produce more value. With this funding, we hope to support more customers and teach the world that there is a better way to build software—feature flagging.

*Header image credit: NASA astronaut Sunita Williams, Expedition 32 flight engineer.

17 Nov 2017

The Only Constant in Modern Infrastructure, is Change

Photo by Federico Beccari on Unsplash

We all know what an audit log is…right? We think of the audit log as a chronological list of changes in a system. Martin Fowler defines it as:

“…the simplest, yet also one of the most effective forms of tracking temporal information. The idea is that any time something significant happens you write some record indicating what happened and when it happened.”¹

While event logs will tell you that a thing happened, the audit log should also tell you what happened, when it happened, and who set that event into motion.

A Practice, Not a Product

I’m sure you can appreciate why having this level of detail is important. By understanding what, when, and who impacted an event that occurred in your system, you now have a timeline of how things have been changed, can infer why the change happened, and decide what your next steps should be.

A great example of this is with security and compliance. It’s important to show a record of what transpired and have accountability. But the audit log is a practice, not a product. You have to think about how you track and record this information so that you can access it later.

One of the things that distinguishes LaunchDarkly from homegrown feature flagging systems, is that often people who build their own don’t take the time to build an audit log. This is usually because an audit log requires a role based auth system that will allow you to track accountability. A standard database doesn’t inherently have a way to track accountability, you must add that to the schema. LaunchDarkly, on the other hand,  enforces this by requiring all users to authenticate into the system, even API access is authenticated and scopes through the use of  personal access tokens.

Let’s Play ‘What-If?’

Your team has a new feature, and you’re releasing it to your end users. You’ve decided to do a percentage rollout because you’re also testing in production—you want to be sure this new feature isn’t going to have a negative impact on any customers. You’ve also explicitly excluded some specific users. You know they’re in a quiet period, and this new feature would not be beneficial to them right now. Maybe you’ll roll this out to them in two weeks when they’re ready.

Just when you think the rollout is going smoothly, you get a call from a customer. They’ve reached out to support wanting to know what’s going on. “There’s a new feature and it’s impacting our performance!” How do you find out what happened?

What if you didn’t have an audit log? Well this process would look very different. This would be a lengthy process in which you (and your team) must look through everything to try and figure out what happened. What flag impacted the change? What was deleted or changed? Do you even have a dashboard to help you review this information? Who made the change—was it on the engineering side or the product team? How many people in support, product, and engineering do you need to talk to to sort out what could be the cause of this error? And keep in mind, you’re probably relying on people’s memory of what they might have done with 100s of flags… This is not an easy task!

The More You Know

But if you did have an audit log in place, you could quickly go back into your records and identify where a change was made, what the change was, and who made it. In this case, someone thought the rollout was doing so well, they decided to push it to 100%. However, they didn’t realize there were customers excluded from the rollout because they shouldn’t be experiencing this kind of change just yet. You could see that the exclusion rule was deleted, and you now know what you can do to fix this and move forward.

So, now I’m sure you can appreciate the value of an audit log over a simple event log. And what your team could be doing instead of hunting down elusive unknowns and what-ifs. With this valuable knowledge in hand, you can quickly identify why an event happened, and decide on how to proceed next with more confidence.

¹Martin Fowler (2004), The Audit Log

13 Nov 2017

To Be Continuous: Transforming Microsoft Into An Open Source Company

In this episode of To Be Continuous, Edith and Paul are joined by Keith Ballinger and Thomas Dohmke from Microsoft. They share their experience of being acquired by Microsoft and discuss the role of DevOps in the continuous delivery process. They then consider the importance of balancing founder convictions of product value and how the world is changing, with necessary market validation activities. Keith also shares his thoughts on why early-stage founders should ignore larger companies for as long as possible, focusing instead on building their business.

This is episode #40 in the To Be Continuous podcast series all about continuous delivery and software development.

Continue reading “To Be Continuous: Transforming Microsoft Into An Open Source Company” »

09 Nov 2017

Measure Twice, Launch Once

You want all your developers to have access to the main trunk of code to deploy — that’s the point of trunk-based development. It’s important they can put their code out as often as they want and iterate on their projects. However, you don’t always want developers turning on features that will have customer impact without some way to reverse course.

Secured activation is an under-appreciated part of feature management. Your developers can deploy code whenever they want—but when it comes time to test it externally, or turn it on for everyone, you can use settings to make sure that only a select group of people has the permissions to do so. All the activation changes should be tracked and audited to ensure that all activations have an accountability chain.

At LaunchDarkly, we have found that it’s good to be permissive about who can use and create feature flags, and restrictive about who can activate them. If you are trying to get started with transitioning to using feature flags more broadly, you might want to think about how to implement a repeatable process. You might also want to leverage LaunchDarkly’s ‘Tags’ feature to help with the organization and custom roles to assist with delegation and access.

You want the following qualities in people who have the permissions to change your user experience:

  • Understands the business reason for making the change
  • Has the technical knowledge or advisors to know when the code is ready to go live
  • Has a process in place for making the change and then testing it

You don’t want to have only one person who can do this, because they’ll inevitably become a bottleneck. Make sure your process can keep releasing even if a key team member is unavailable.

In the beginning you may look to put a process around every change and then look for optimization in that process. However, over time you should look into determining  what level of change merits process and what can be executed more easily. In some cases this might even allow for small changes to be approved or executed by individual engineers. Usually, features that have anything to do with money, user data collection, or changes in the user process should have a formal approval process. Changes to backend operations can be quieter and therefore need less formal process and lean more heavily on automated testing and peer review.

Think about your current deployment process. What happens if someone releases something too early? How do you protect against that? How will you port that control over to the access control that LaunchDarkly offers? What is the failure case if something doesn’t launch properly?

Feature flags are easy to implement in code, but managing them well across an organization takes some planning and forethought.

18 Aug 2017

Risk Reduction and Harm Mitigation

Risk Reduction is trying to make sure bad things happen as rarely as possible. It is anti-lock brakes, vaccinations, clothing irons that turn off by themselves, and all sorts of things that we think of as safety modifications in our life. We are trying to build lives where bad things happen less often.

Harm Mitigation is what we do to make sure that when bad things do happen, they are less catastrophic. Fire sprinklers in buildings, seat belts, and needle exchanges are all about making the consequences of something bad less terrible.

What does that mean for a DevOps world where our risks and harms are very different? Charity Majors says we shouldn’t split the world into developers and operations, but into product and infrastructure—and I think that’s a useful way to think about risk too.

Product risk is problems that users experience. We can usually predict and mitigate the danger by testing and being aware of common failings. For example, we can expect and plan for users that may be on a flaky connection, or that they may try to exit a page without saving information. We can work around these problems because we know they’re out there. But our deployments are not something they should see until we’re ready for prime time. For this kind of risk, we use feature flags to control what content is delivered.

Infrastructure risk is more about the inherent fragility of delivering software. CDNs, fiber, servers, switches, towers…the whole system of getting data to people has failure zones. When we are trying to reduce infrastructure risk, we assume that latency is ever-present, networks are intermittent, and we won’t always be able to count on everything working right the first time. We try to build in robust and flexible ways than can route around failures. This is the place we might use feature flags to control failovers or to create circuit breakers to prevent flooding a fragile sector.

Product harm reduction is about making sure that users can have a positive experience, even if something happens that makes it less than ideal. We want to preserve their blog drafts, keep them from committing errors, only show them things they are allowed to change, and above all, avoid giving them a blank page. Something has gone wrong in the trip to the user, but they shouldn’t have to suffer for it.

Infrastructure harm reduction is the ability to pull back breaking fixes, shunt users away from vulnerabilities, and respond near-immediately to things that have gone very wrong. Harm reduction at this level is the kind of action a pager-responder can take to get things back on track before doing more intensive repairs and investigations in the morning.

In my first week, I spent a lot of time thinking about how to summarize our product for a variety of audiences. “Feature flags as a service” is short and pithy, but only works if you can bring your own definition of “feature flag” and a business understanding of why you would use them. What about “Feature flags allow you to make changes in near real-time instead of waiting for a deployment, and LaunchDarkly helps you manage and track flags across an organization”?

Well, that works, but it still doesn’t get to the core of why an organization would want to use feature flags. What I’ve come up with so far is this:

Feature flags segment the risk of creating a product into manageable parts.

Creating and deploying software is risky. We can accidentally build in errors, we can deliver it badly, or to the wrong people, and it can interact in unfortunate ways with existing software or hardware. As organizations, we want to do our best to do no harm and provide benefit. Using feature flags lets us wrap our features in decision points that we can then use to make life easier for our users.

Here are some types of risks that are reduced by using feature flags:

  • Server falls over from too much traffic
  • Canary launch is not well-tracked, problems are missed
  • Old features and workarounds are invisible and get left in place
  • Feature with vulnerable content is deployed
  • API endpoints are exposed to unauthorized users

Managing your feature flags is a post for another day—today I ask that you take a few minutes and think about how you can reduce risk and minimize harm in your organization, your project, or your code. How can you make things robust enough to resist failure, instrumented sufficiently to identify a failure spot, and flexible enough to reduce harmful consequences on the fly?

08 Feb 2017

Toggle Talk with Slack’s Director of Engineering Josh Wills

I sat down with Josh Wills, Director of Engineering at Slack and unabashed feature flag enthusiast, to get his opinions on the practice, where it’s most useful, and how it has played an important part in his career.  I think my favorite take-away from him was,

“Feature flagging is scary. I get why it’s scary. But to me, not launching your product every five minutes is scary… Launching continuously is how you learn fast– It’s not just about deploying fast, it’s about learning fast. That’s the future of viability.”

Here’s what I heard from Josh in the interview.  

 

How long have you been feature flagging?

I started feature flagging at Google in 2007, and when I joined the team they were in the middle of rewriting the feature flag system. Our feature flag framework was called the Experiments Framework because it was designed for running A/B tests, and it grew out of that into this very powerful and complicated feature flagging framework that we used for literally everything. It started out as a simple “feature on”/“feature off” system, but it evolved to include richer types like strings and floats and even had some simple conditional logic for modifying flags based on attributes of the request. It became the system through which all of Google’s various machine learning models were combined to make decisions for ad ranking, search ranking, etc.

Imagine that you have dozens of machine learning models active on a given request, doing all kinds of different things, and your job is to figure out an algorithm for deciding which ads should go where and how much each advertiser needs to pay. All of those different signals are combined together through a complicated series of equations which have a bunch of thresholds and weights. There’s very rich logic in not only the machine learning models, obviously, but also within the feature flag framework, to control under different contexts what counts the most.

Trying to do data science and machine learning in production without feature flags is nuts.

The feature flag framework here at Slack was developed in-house and was not initially developed with data science in mind, but we eventually created one that was to support our own search ranking, backend performance, and new team onboarding experiments.

But when I got here, I was so happy that they had a feature flag framework at all– it means you deploy code hundreds of times a day, not once a month, or whatever it is other companies do.

 

What do you prefer to call it and why?

I like to call it the experiment framework, but that’s just Google/Xoogler nomenclature. Facebook’s system is called Gatekeeper and it’s basically the same idea. Most of these systems have converged to have the same set of features, because it just makes sense and you have to have certain things. Eventually you’ll get there anyway, so why not just skip to the end?  

 

When do you think feature flagging is most useful?

I’m biased, since I’m a data person. Data science, machine learning is what I do and I think it is absolutely critical for machine learning, for bringing any kind of data driven feedback loops and intelligence into your application. That is when it is absolutely most critical.

You should not be doing machine learning without a feature flag framework.

If you are saying, “Oh I want to get into machine learning, and I’m going to do predictions or personalization or recommendations,” which lots of people do, and you are doing it without a feature flag framework, you are insane and should be fired. Not to put too fine a point on it, but…

 

Are there any cases where feature flagging is not a good idea?

Well, there is always tech debt that goes along with doing this kind of stuff. We deal with this at Slack, and Google deals with it as well. You end up with code that has a lot of “if” statements in it. Engineers are not always the best about deleting their feature flags once they’re no longer necessary.

So we have a whole archival system, so that when you do the code review to create the feature flag, you also have to specify when you plan to delete the flag, and get alerts if the flag is not removed by such-and-such a date. That is the cost of doing business. I would say the benefits massively outweigh the costs.

I think there are certain other situations where feature flags are not a good idea.  They’re relatively rare, but they do happen. Where you’re switching backend systems there can be times where you have to just go for it. You maybe reach a point where you just can’t quite bring yourself to burn the bridge and just go without the old system anymore, even though it’s causing you pain, and you know it needs to go. You just need to bite the bullet and do it, do the cut over and live with the consequences.

You work super hard to not ever find yourself in that situation. No team is going to go out of their way to corner themselves like that, but if you’re cornered, you’ve got to fight your way out.

 

Best use of a feature flag – a personal story?

It’s been crucial to the way I’ve worked ever since I moved to San Francisco. Coming from the perspective of doing machine learning, data science, etc. at Google, I ran thousands of experiments all in search of new ways of combining different models together, in order to fundamentally make Google more money, make the user experience better, and generate more ROI for advertisers. I got really good at it. You could say that feature flags made my career.

At the same time, there was one Friday afternoon back in 2009 where I thought it would be a good idea to do one last push, and I launched a bad feature flag configuration that broke the ads systems and cost Google a lot of money. I remember my boss saying at the time, “So Josh, what did you learn from the X million dollars we just spent educating you?”

It’s the kind of thing where you learn that canaries are good, and that 5 pm pushes on a Friday are pretty much always a bad idea, no matter how good your feature flagging framework is.

 

What do you think is the number one mistake that’s made around feature flagging?

I think my biggest thing is if you are really thinking of feature flags as just “on” “off” switches, you are missing the real power of what they can do.

To me the feature flag space is the parameter space that I get to explore to optimize whatever metrics that I want to optimize. If you are constraining yourself to this very limited boolean on/off space without strings, floats, etc. you’re putting artificial limits on how fast you can explore the space and about how all of the knobs at your disposal work.

This will be the early mistake that I think a lot of people make– they won’t feature flag often enough.

 

Can you share any tips for better flagging?

I think what was most compelling for me at Google was the configuration language for feature flags was very rich, but not Turing complete. I don’t think that configuration-as-code is a good idea for feature flags, because it becomes harder to test/validate, which slows down how quickly you can roll things out. However, the configuration language was like programming in the sense that you could define what we called “condition functions” that could be evaluated in the context of a request and used to adjust the values of the flags. So the logic was like a long series of “if” statements, where you could modify the resulting value either by overriding it or by addition, multiplication, or other custom operators.

The way it worked when I was at Google and working on the ad system was that the server binary was pushed weekly, so the only changes that you could make during the week were via experiments. Having that sort of richness of programming via feature flags allowed a lot more freedom and a lot rapid safer iteration between those weekly pushes. The same logic applies today for mobile apps, where you can only do releases via an app store, but you want to be learning what works much faster than that.

 

How do you think feature flags play into the DevOps movement? Continuous Delivery?

How would you do anything without them?

What does it mean to do DevOps without feature flags? It’s one of those things that doesn’t make sense to me. How would you make mayonnaise without eggs? It’s like, not really mayonnaise then. Which is good, because mayonnaise is gross.

I would be fascinated and somewhat horrified to have someone try to explain a feature flag-less DevOps set up. I don’t know what would that look like.

 

Are you seeing feature flagging evolving? If so how? And how do you expect it to change in the future?

Not as much as I would like, broadly speaking. I’m not at Google anymore, so I don’t know what the current state was when I left. The stuff they had is much richer than anything I’ve seen anywhere else, but to be fair, they were doing a lot of machine learning much earlier than anyone else, too.

I think feature flags need to find a way to strike the balance between configuration as on/off switches and configuration as Turing-complete programming language. That was the thing I felt was most compelling and powerful about the way Google did it that I’ve not seen anywhere else.