10 Feb

LaunchDarkly Values

LaunchDarkly Team
  • Respect and integrity for our team, our customers, and our community.
  • We believe in teams, not fiefdoms. Leaders, not tyrants.
  • We’re building a place where you can learn and grow.
  • Work is not life.

When John & I started LaunchDarkly, we’d known each other for years, since being undergraduates at Harvey Mudd College. Part of why we started the company together is that we shared common values about the company that we wanted to work at. As a founder, you’re not only building a product – you’re building a culture.

As we got bigger, it was time to formalize our values from tribal “this is the way we work” knowledge to something more scalable. We surveyed our current team, and I was thrilled that, what I thought our values should be, the team already thought were in place. We reviewed our values at our last team meeting, and everyone gave an example of the time they’ve seen them practiced.

“Values are valuable when you have a hard decision to make,” is John’s and my view. By making our values public, we hope both our customers and our team can know who we are.

08 Feb

And so the adventure begins

It’s something of a LaunchDarkly tradition to write a blog post at the end of your first week. But here I sit, deep into week number two, and I haven’t written a thing. In my defense things move pretty fast around here, and I was just trying to keep my head above water. My second day on the job, having finally set everything up so I could start coding, they had me deploy production. By the time Friday rolled around, I had fixed 3 bugs, opened 6 pull requests, deployed production two more times — and written zero blog posts. Better late than never though, right?

A new chapter

I’m extremely thrilled to be part of this great company. I joined LaunchDarkly after nearly 10 years at Atlassian, where I spent most of my time doing front-end development and arguing about UX. I’m excited to take everything I learned there and continue to make LaunchDarkly a product people love to use.

I took the job because I believed in the people, the product, and the culture. If anything I underestimated all of those things and I can’t wait to see where this journey takes me.

An escape

The other day we went on a team outing to an escape room. If you’re not familiar, you basically solve a series of puzzles using items in a room in order to “escape” the room. I’ve never done one before, so I have nothing to compare it to, but this particular escape room required a lot of coordinated effort. We had 12 people participating (almost the whole team, which is at 15 now), but at any given time there were probably 3-5 puzzles being worked on.

The beginning was a little chaotic as everyone tried to figure out exactly what we’d gotten ourselves into and what in the world to do next. But pretty soon we settled down, started self-organizing, and talking through the problems. It was pretty amazing to see everyone working together so well — no arguments or drama, no jockeying for leadership, just good old-fashioned teamwork. I think that experience sums up my time at LaunchDarkly so far — fast and fun, with a great group of people working together towards a common goal.

Oh yeah, and we escaped the room (just barely).

08 Feb

Toggle Talk with Slack’s Director of Engineering Josh Wills

I sat down with Josh Wills, Director of Engineering at Slack and unabashed feature flag enthusiast, to get his opinions on the practice, where it’s most useful, and how it has played an important part in his career.  I think my favorite take-away from him was,

“Feature flagging is scary. I get why it’s scary. But to me, not launching your product every five minutes is scary… Launching continuously is how you learn fast– It’s not just about deploying fast, it’s about learning fast. That’s the future of viability.”

Here’s what I heard from Josh in the interview.  

 

How long have you been feature flagging?

I started feature flagging at Google in 2007, and when I joined the team they were in the middle of rewriting the feature flag system. Our feature flag framework was called the Experiments Framework because it was designed for running A/B tests, and it grew out of that into this very powerful and complicated feature flagging framework that we used for literally everything. It started out as a simple “feature on”/“feature off” system, but it evolved to include richer types like strings and floats and even had some simple conditional logic for modifying flags based on attributes of the request. It became the system through which all of Google’s various machine learning models were combined to make decisions for ad ranking, search ranking, etc.

Imagine that you have dozens of machine learning models active on a given request, doing all kinds of different things, and your job is to figure out an algorithm for deciding which ads should go where and how much each advertiser needs to pay. All of those different signals are combined together through a complicated series of equations which have a bunch of thresholds and weights. There’s very rich logic in not only the machine learning models, obviously, but also within the feature flag framework, to control under different contexts what counts the most.

Trying to do data science and machine learning in production without feature flags is nuts.

The feature flag framework here at Slack was developed in-house and was not initially developed with data science in mind, but we eventually created one that was to support our own search ranking, backend performance, and new team onboarding experiments.

But when I got here, I was so happy that they had a feature flag framework at all– it means you deploy code hundreds of times a day, not once a month, or whatever it is other companies do.

 

What do you prefer to call it and why?

I like to call it the experiment framework, but that’s just Google/Xoogler nomenclature. Facebook’s system is called Gatekeeper and it’s basically the same idea. Most of these systems have converged to have the same set of features, because it just makes sense and you have to have certain things. Eventually you’ll get there anyway, so why not just skip to the end?  

 

When do you think feature flagging is most useful?

I’m biased, since I’m a data person. Data science, machine learning is what I do and I think it is absolutely critical for machine learning, for bringing any kind of data driven feedback loops and intelligence into your application. That is when it is absolutely most critical.

You should not be doing machine learning without a feature flag framework.

If you are saying, “Oh I want to get into machine learning, and I’m going to do predictions or personalization or recommendations,” which lots of people do, and you are doing it without a feature flag framework, you are insane and should be fired. Not to put too fine a point on it, but…

 

Are there any cases where feature flagging is not a good idea?

Well, there is always tech debt that goes along with doing this kind of stuff. We deal with this at Slack, and Google deals with it as well. You end up with code that has a lot of “if” statements in it. Engineers are not always the best about deleting their feature flags once they’re no longer necessary.

So we have a whole archival system, so that when you do the code review to create the feature flag, you also have to specify when you plan to delete the flag, and get alerts if the flag is not removed by such-and-such a date. That is the cost of doing business. I would say the benefits massively outweigh the costs.

I think there are certain other situations where feature flags are not a good idea.  They’re relatively rare, but they do happen. Where you’re switching backend systems there can be times where you have to just go for it. You maybe reach a point where you just can’t quite bring yourself to burn the bridge and just go without the old system anymore, even though it’s causing you pain, and you know it needs to go. You just need to bite the bullet and do it, do the cut over and live with the consequences.

You work super hard to not ever find yourself in that situation. No team is going to go out of their way to corner themselves like that, but if you’re cornered, you’ve got to fight your way out.

 

Best use of a feature flag – a personal story?

It’s been crucial to the way I’ve worked ever since I moved to San Francisco. Coming from the perspective of doing machine learning, data science, etc. at Google, I ran thousands of experiments all in search of new ways of combining different models together, in order to fundamentally make Google more money, make the user experience better, and generate more ROI for advertisers. I got really good at it. You could say that feature flags made my career.

At the same time, there was one Friday afternoon back in 2009 where I thought it would be a good idea to do one last push, and I launched a bad feature flag configuration that broke the ads systems and cost Google a lot of money. I remember my boss saying at the time, “So Josh, what did you learn from the X million dollars we just spent educating you?”

It’s the kind of thing where you learn that canaries are good, and that 5 pm pushes on a Friday are pretty much always a bad idea, no matter how good your feature flagging framework is.

 

What do you think is the number one mistake that’s made around feature flagging?

I think my biggest thing is if you are really thinking of feature flags as just “on” “off” switches, you are missing the real power of what they can do.

To me the feature flag space is the parameter space that I get to explore to optimize whatever metrics that I want to optimize. If you are constraining yourself to this very limited boolean on/off space without strings, floats, etc. you’re putting artificial limits on how fast you can explore the space and about how all of the knobs at your disposal work.

This will be the early mistake that I think a lot of people make– they won’t feature flag often enough.

 

Can you share any tips for better flagging?

I think what was most compelling for me at Google was the configuration language for feature flags was very rich, but not Turing complete. I don’t think that configuration-as-code is a good idea for feature flags, because it becomes harder to test/validate, which slows down how quickly you can roll things out. However, the configuration language was like programming in the sense that you could define what we called “condition functions” that could be evaluated in the context of a request and used to adjust the values of the flags. So the logic was like a long series of “if” statements, where you could modify the resulting value either by overriding it or by addition, multiplication, or other custom operators.

The way it worked when I was at Google and working on the ad system was that the server binary was pushed weekly, so the only changes that you could make during the week were via experiments. Having that sort of richness of programming via feature flags allowed a lot more freedom and a lot rapid safer iteration between those weekly pushes. The same logic applies today for mobile apps, where you can only do releases via an app store, but you want to be learning what works much faster than that.

 

How do you think feature flags play into the DevOps movement? Continuous Delivery?

How would you do anything without them?

What does it mean to do DevOps without feature flags? It’s one of those things that doesn’t make sense to me. How would you make mayonnaise without eggs? It’s like, not really mayonnaise then. Which is good, because mayonnaise is gross.

I would be fascinated and somewhat horrified to have someone try to explain a feature flag-less DevOps set up. I don’t know what would that look like.

 

Are you seeing feature flagging evolving? If so how? And how do you expect it to change in the future?

Not as much as I would like, broadly speaking. I’m not at Google anymore, so I don’t know what the current state was when I left. The stuff they had is much richer than anything I’ve seen anywhere else, but to be fair, they were doing a lot of machine learning much earlier than anyone else, too.

I think feature flags need to find a way to strike the balance between configuration as on/off switches and configuration as Turing-complete programming language. That was the thing I felt was most compelling and powerful about the way Google did it that I’ve not seen anywhere else.

 

03 Feb

My First Week

Start your Engines

As I reminisce on my first week at LaunchDarkly, I’m reminded about the initial call I had with Russ, our VP of Sales, to learn more about the company and opportunity. Russ and I have known each other since 2012, when I interviewed for his team at Box. Although we didn’t end up working together, we kept in touch and kept joking that one day, we would! I guess you could say it was fate that caused that call a few months ago to happen…or just perfect timing.

When I spoke to Russ and learned more about LaunchDarkly, I immediately felt that gut instinct say “yes”. It was clear to me that this represented a great opportunity to join an awesome team that had built an amazing product that customers loved. Now, I look back on my first week as validation for the gut feeling I first had.

Shake and Bake

The whole week was filled with highlights. Monday started out with our first revenue meeting, then an all hands where John and Edith walked us through the company values (which I agreed with wholeheartedly). I sat in on a variety of customer calls, which showed me how interested companies are to speak with us and how valuable our platform is to them. The sales team met with one of our advisors (a former co-founder / CTO) and had a great meeting learning about how best to partner with VPs of Engineering / CTOs. And then on top of that, I learned more about the product, the market opportunity, and the company vision.

What I’m left with after that first week is being continually impressed by the company and product John, Edith, and the team have built. From hearing successful customer stories, meeting other team members, and learning about the strength of the product, I’m filled with confidence in our ability to succeed. More so, I’m excited to start partnering with companies and help them deliver better software, faster.  It’s been a great first week, and I’m looking forward to many more.

01 Feb

Leading Feature Flag Platform LaunchDarkly announces industry topping NPS

I’m thrilled that LaunchDarkly, the leading feature flag management platform, has a Net Promoter Score of 50! Net Promoter Score is a measure of how much customers are willing to recommend a product. 30 is an average score for a tech vendor, so to be in top tier of tech vendors is wonderful validation for us. But the NPS score is just a scalar. What I really enjoyed hearing was customer appreciation of how LaunchDarkly’s feature flag platform was easy to use, reliable and powerful. We help less risky better releases, allowing business users and developers to collaborate for an overall transformative experience. LaunchDarkly customers are happy that they don’t have to build a feature flagging platform in house. Don’t take my word for it – here’s what our customers have to say:

Transformative

  • “Feature flags are the future, you help reduce the barrier-to-entry.”
  • “It’s amazing – changed the way we release new features.”
  • “It’s cool tech that works well. Encourages good develop and release practices.”
  • “Launch Darkly is a really good service and should be used for good software practices.”

Ease of Use

  • “Very convenient and fast way to roll-out new feature to users.”
  • “Very easy to use. Easy to setup targets. Easy to roll out and rollback if needed.”
  • “It offers more flexibility and ease of toggling new/development features with the rollout and inclusion/exclusion features than our previous solution.”
  • “Much better way to introduce a new or changed functionality, compared to configuration changes”
  • “Easy to use. Great UI”
  • “Really like how configurable things are in LD, esp access controls, etc. Very easy to use”

Reliable and Powerful

  • “It’s an incredibly powerful feature flagging system, the best I have ever come across. So far it the uptime, and performance is perfect.”
  • “Its awesome and is a key component in hitting uptime targets”
  • “so easy to turn on and off functionality . Efficient, performance – wise.”
  • “The service has been really reliable.”
  • “We’ve had a case where a third party failed and was causing our systems to crash. Lucky we had a feature flag that covered that part of the code and was able to turn it off until the problem was resolved.”
  • “Instant availability of changed/created flags. Super easy to create and change flags.”

Less Risky, Better Releases

  • “I like the concept of using feature flags as it allows us to ship new features without having them complete. I’ve already recommended LaunchDarkly to a few of my developer friends. :)”
  • “It has made releasing new features less stressful and made the release of features along with training and marketing materials a snap”
  • “The ease of use through the user interface of the exclusion and inclusion is really helpful for my day-to-day. I can easily set up two different users, environments, etc. to test against a feature on vs. off. It also makes the release process less stressful!”

Buy vs Build

  • “Feature toggling is critical and nobody should be writing code to solve a solved problem.”
  • “Great tool replacing custom flagging mechanism, cost effective.”
  • “It is easy to use, and way better than having to implement your own functionality”
  • “It’s cheaper than developing the capability yourself. Generally if something is not core to your business or something that differentiates you in the marketplace then I think you should try to outsource it to a 3rd party. Feature toggling is an excellent example of that.”
  • “Provides all the functionality plus more that teams internally were building themselves”
  • “It is a better “buy” of feature flags than any of us would reasonably “build”.”

Business Users

  • “It allows us to offload management of which features are exposed to whom to our customer success team, so the engineering organization doesn’t have to make a change in our database or build our own internal launch darkly.”
  • “Saves our devs time and allows me to quickly tailor settings before demos, trainings, etc.”
  • “Has completely changed how we’re able to manage our release process. We now implement flags on each new feature and we can keep deploying as long as the build is green, the product/marketing teams are then responsible for launches once they’re satisfied.”
  • “Controlling features on the dashboard is incredibly easy for engineers to non-tech people”
  • “Was really simple to get the developers to adopt, almost as frictionless to explain the concepts to the product/marketing teams.”

As CEO & founder of LaunchDarkly, I love passing along kudos to our team:

  • “Thanks for making such an awesome product!”
  • “Keep up the good work. Keen to see what you’ve got coming up.”
  • “keep up the good work :)”
  • “Awesome job!”
  • “keep doing what you’re doing’
  • “LD seems like _such_ a small thing, but I think you’ve made a really good product”

As good as 50 NPS is, we still have room to improve, so we’ve set a target for 60. I’ve personally read all your feedback, and we’re looking forward to improving so our customers are even more likely to recommend us. Feel left out? Want to make your voice heard? Look in your email for your own invitation to our NPS survey – I really do read every one.

01 Feb

To Be Continuous: What’s The Future of Continuous Delivery?

In this episode, Edith and Paul are joined by Marianna Tessel, VP of Engineering at Docker to discuss the evolution of software delivery from the days of punch card systems. Marianna shares her thoughts on the future of continuous delivery, where she envisions software will be written in small chunks, instantly sucked into the system and ready to be used.

Marianna also shares lessons from her days at VMware on selling to enterprise clients and driving bottoms-up developer adoption.

Is ‘next shiny thing syndrome’ keeping you from being successful? Listen in to find out more.

This is episode #29 in the To Be Continuous podcast series all about continuous delivery and software development.

Continue reading “To Be Continuous: What’s The Future of Continuous Delivery?” »