31 Aug 2016

Secrets of Netflix’s Engineering Culture

Netflix is not just known for the cultural phenomena of “Netflix and chill”, but for its legendary engineering team that releases hundreds of times a day in a data-driven culture. Netflix is the undisputed winner in the video wars, having driven Blockbuster into the “return” bin of history. Netflix won by iterating quickly and innovating with numerous micro-deployments. Could what worked for Netflix work for you?

Netflix had a virtuous cycle of product innovation. Every change made in the product is with the goal of getting new users to become subscribers. Netflix has a constant flow of new users every month, so they always have new users to test on. Also, they have a vast store of past data to optimize on. Did someone who liked “Princess Bride” also like “Monty Python and the Holy Grail”? When is the right time to prompt for a subscription? Interesting tests that Netflix can run include whether TV ads drive Netflix signups, or whether requiring Facebook to create an account drives enough social activity to counteract the drop in subscriptions from people who don’t have Facebook. If a change increased new user subscriptions, it went into the product. If it didn’t increase new user subscriptions, it didn’t make it in – hypothesis driven development.

However, what if you’re not Netflix? What if you’re a steady SaaS business with 1,000 business customers, on boarding 30 new customers a month? This is a healthy business, doubling in size annually. However what if you wanted to test whether you get more subscriptions with a one step or two step process to add a credit card. With a sample set of 30 a month & 90% current success rate, it will take you three months to determine success. Not everything can be tested at small scale. Tomasz Tunguz talks more about the perils of testing early here.

The other “gotcha” to watch out for with Netflix style development is obsessive focus on one metric can degrade other metrics. For example, focusing on optimizing new user signup might mean degrading experience for old users. Let’s say that 10,000 customers could be served with “good” speed, or 2,000 with “superfast” speed and 8,000 with “not good speed”. Or 1,000 with lightning fast and 9,000 with terrible speed. You might make the 1,000 new customers very happy, but piss off the 9,000 existing customers and have them quit. A good counterweight is to always have a contra-metric to keep an eye on. It’s okay if it dips slightly if the main metric rises. However, if the other metric tanks, re-consider whether the overall gains are worth it.

So what lessons can you take from Netflix to help your own business?

One, have a clear idea of why you’re making changes, even if it’s not something that you can a/b test. Is it to increase stability in your system? Make it quicker for someone to onboard? Know what your success criteria are, even if there’s not a statistically significant “winner”.

Two, break down projects into easily quantifiable chunks of value. Velocity can be as important (if not more important) than always being right. For example if you try 20 small changes, and half are right, you’ll end up 50% better. If you try one big change, and it’s not accretive, you’ll end up with a zero percent gain. Or, as Adrian Cockcroft, Netflix Architect says “If you’re doing quarterly releases and your competitor is doing daily releases you will fall so far behind”.

Three, don’t underestimate the importance of your own domain expertise. If you’re constantly testing ideas, even without having enough data, you’re quicker to get into the right path. Let your competitors copy your past mistakes, while you move forward. As Kris Gale, co-founder and CTO of Clover Health said, “You will always make better decisions with more information, and you will always have more information in the future.” But the way to get more information is to iterate.



26 Aug 2016

3 Ways to Avoid Technical Debt when Feature Flagging

Feature flags are a valuable technique of separating out release (deployment) from visibility. Feature flags allow a software organization to turn features on and off at a high level, as well as segment out their base to allow different users different levels of access. However, feature flags have an (ill-deserved) reputation of “Technical Debt”. Used incorrectly, feature flags can accumulate, add complexity, and even break your system. Used correctly, feature flags can help you move faster. Here’s three easy ways you can avoid technical debt when using feature flags.

  1. Create a central repository for feature flags

Using config files for feature flags is “the junk drawer” of technical debt. If you have seven config files with different flags for different parts of the system, it’s hard to know what flags exist, or how they interact. Have one place where you manage all of your feature flags.

  1. Avoid ambiguously named flags

Give your flags easy to understand, intuitive names. Assume that someone other than you and your flag could potentially be using this flag days, months, and years into the future. Don’t have a name that could cause someone to turn it on when they mean off, or vice versa. For example “FilterUser”, when it’s off – does this mean users are filtered? or not?

  1. Have a plan for flag removal

Some flags are meant for permanent control, for example for an entitlements system. Other flags are temporary, meant for the purpose of a release only. If a flag can be removed (because it’s serving 100% or 0% of traffic), it should be removed, as quickly as possible. To enforce this rigor, when you write the flag, also write the pull request to remove it. That way, when it’s time to remove the flag, it’s a two second task.

22 Aug 2016

Feature flagging and testing

Sam Stokes, Rapportive co-founder, has this tip on testing and feature flagging:

“I’ve found that explicitly testing both sides of the flag can help with managing the increased complexity a flag introduces.  It helps you catch changes down the line that would have broken the old, forgotten code path that 10% of customers are still on.  It provides reminders to clean up the old code and delete the flag (via occasionally breaking the tests).  And it provides some pushback against having too many interacting flags (if you had to write 8 tests for one function because it depended on three separate feature flags, maybe it’s time to refactor that function, or retire some flags).”

Sam Stokes regularly blogs here.

17 May 2016

How to use feature flags without technical debt

Or, won’t feature flags pollute my code with a bunch of crufty if/then branches that I have to clean up later?

People often ask for advice on maintaining feature flags after they roll out the new code to all users. Here is one way to approach the issue of cleaning up the old code paths. It works pretty well for our team. If you have any other ideas, please let us know!

Write the new feature

Step one is to write the new feature, gated by the feature flag, on a short-lived feature branch off of master (call it pk/awesome-xyz-support):

Next, before you submit your pull request for this change, create a second branch, off of the first, and call it cleanup/awesome-xyz. This branch removes the feature flag, leaving just the new code:

Continue reading “How to use feature flags without technical debt” »

13 May 2016

LaunchDarkly – Now Serving Billions of Feature Flags at Warp Speed

It’s been a busy spring for LaunchDarkly, the leading feature flag management company.  We’ve hit a huge milestone – we’re serving  Billions (with a B) feature flag events daily, for companies all over the world. 

LaunchDarkly helps companies with better DevOps by managing code deployment separately from business logic. We allow an entire product development organization to control who gets the right feature, when. Developer tool companies like CircleCI, Apiary and AppDirect use us to power their own DevOps.  But, LaunchDarkly isn’t just for developer tool companies to reduce risk and iterate quickly on feature release.  Across industries, companies like Ten-X, Behalf, Handshake, and TrustPilot integrated LaunchDarkly into their development and release cycles to move faster with higher quality. 

Why are companies moving so quickly to LaunchDarkly?  We offer true polyglot support for web and mobile applications. Have a Node.js web application with Go micro-services and a mobile app written in Swift?  We got you covered.  And, more importantly, we allow companies like CircleCI and Behalf to go beyond DevOps to transform the way they deliver software from idea to delivery by uniting product management, engineering, QA and operations. We offer end to end feature flag management to allow an entire product development organization to control who gets the right feature, when.

To help spread the word about the power of feature flagging, we also have awesome partners.  We were thrilled to participate in the Microsoft Build in March.  I was part of the Microsoft DevOps Keynote, showing our end to end integration with Visual Studio Team Services.  And we’re proud to power feature rollouts for effective DevOps at companies like: Continue reading “LaunchDarkly – Now Serving Billions of Feature Flags at Warp Speed” »

26 Apr 2016

To Be Continuous: DevOps 2.0

In this episode, Edith and Paul discuss the emergence of the term DevOps 2.0 and try to decide what it means, they talk about the incredible early validation powers continuous delivery gives your team, and they denounce the unnecessary risks of April Fools jokes not backed by continuous delivery. This is episode #18 in the To Be Continuous podcast series all about continuous delivery and software development.

Continue reading “To Be Continuous: DevOps 2.0” »