07 Sep

Why microservices need feature flags

Microservices is the practice of breaking up a huge, monolithic release where all components are tested and released as a whole, into many discrete services that can go on independent release schedules. The benefits of microservices were popularized by Martin Fowler, and put into practice by Amazon and Netflix. However, with microservices, releases become more complicated, by n-factorial. Instead of one monolith that can tested as a whole, if there are even as few as nine different services, each needs to be tested with every other component. Suddenly there are nine potential friction points.

Feature flags to the rescue! With feature flags, engineering teams can have complete control over their various microservices. First, wrap the microservice with a feature flag, with all traffic going to the old version. Then, release a new version. With a feature flag, gradually put whatever traffic you want to the new version, verifying functionality in all the other micro services. The new traffic and be cordoned off however suits your business. A feature flag at it’s simplest can be an “on” “off” switch to quickly flip between versions, similar to a blue green deployment. However, feature flags can serve arbitrarily complex (or simple) variations of traffic to the new microservice. Some example of how to test the new micro service are
– percentage rollout
– by IP address of service
– by version number of other services
or whatever other information. Feature flags allow users to do “microdeployments” of microservices. A microdeployment is breaking a deployment into smaller components of who exactly receives the new service.

Using feature flags and microservices together allows engineering teams to truly unlock the value of decoupling. You can read more about how we use microservices and feature flags at LaunchDarkly here.




31 Aug

Secrets of Netflix’s Engineering Culture

Netflix is not just known for the cultural phenomena of “Netflix and chill”, but for its legendary engineering team that releases hundreds of times a day in a data-driven culture. Netflix is the undisputed winner in the video wars, having driven Blockbuster into the “return” bin of history. Netflix won by iterating quickly and innovating with numerous micro-deployments. Could what worked for Netflix work for you?

Netflix had a virtuous cycle of product innovation. Every change made in the product is with the goal of getting new users to become subscribers. Netflix has a constant flow of new users every month, so they always have new users to test on. Also, they have a vast store of past data to optimize on. Did someone who liked “Princess Bride” also like “Monty Python and the Holy Grail”? When is the right time to prompt for a subscription? Interesting tests that Netflix can run include whether TV ads drive Netflix signups, or whether requiring Facebook to create an account drives enough social activity to counteract the drop in subscriptions from people who don’t have Facebook. If a change increased new user subscriptions, it went into the product. If it didn’t increase new user subscriptions, it didn’t make it in – hypothesis driven development.

However, what if you’re not Netflix? What if you’re a steady SaaS business with 1,000 business customers, on boarding 30 new customers a month? This is a healthy business, doubling in size annually. However what if you wanted to test whether you get more subscriptions with a one step or two step process to add a credit card. With a sample set of 30 a month & 90% current success rate, it will take you three months to determine success. Not everything can be tested at small scale. Tomasz Tunguz talks more about the perils of testing early here.

The other “gotcha” to watch out for with Netflix style development is obsessive focus on one metric can degrade other metrics. For example, focusing on optimizing new user signup might mean degrading experience for old users. Let’s say that 10,000 customers could be served with “good” speed, or 2,000 with “superfast” speed and 8,000 with “not good speed”. Or 1,000 with lightning fast and 9,000 with terrible speed. You might make the 1,000 new customers very happy, but piss off the 9,000 existing customers and have them quit. A good counterweight is to always have a contra-metric to keep an eye on. It’s okay if it dips slightly if the main metric rises. However, if the other metric tanks, re-consider whether the overall gains are worth it.

So what lessons can you take from Netflix to help your own business?

One, have a clear idea of why you’re making changes, even if it’s not something that you can a/b test. Is it to increase stability in your system? Make it quicker for someone to onboard? Know what your success criteria are, even if there’s not a statistically significant “winner”.

Two, break down projects into easily quantifiable chunks of value. Velocity can be as important (if not more important) than always being right. For example if you try 20 small changes, and half are right, you’ll end up 50% better. If you try one big change, and it’s not accretive, you’ll end up with a zero percent gain. Or, as Adrian Cockcroft, Netflix Architect says “If you’re doing quarterly releases and your competitor is doing daily releases you will fall so far behind”.

Three, don’t underestimate the importance of your own domain expertise. If you’re constantly testing ideas, even without having enough data, you’re quicker to get into the right path. Let your competitors copy your past mistakes, while you move forward. As Kris Gale, co-founder and CTO of Clover Health said, “You will always make better decisions with more information, and you will always have more information in the future.” But the way to get more information is to iterate.



26 Aug

3 Ways to Avoid Technical Debt when Feature Flagging

Feature flags are a valuable technique of separating out release (deployment) from visibility. Feature flags allow a software organization to turn features on and off at a high level, as well as segment out their base to allow different users different levels of access. However, feature flags have an (ill-deserved) reputation of “Technical Debt”. Used incorrectly, feature flags can accumulate, add complexity, and even break your system. Used correctly, feature flags can help you move faster. Here’s three easy ways you can avoid technical debt when using feature flags.

  1. Create a central repository for feature flags

Using config files for feature flags is “the junk drawer” of technical debt. If you have seven config files with different flags for different parts of the system, it’s hard to know what flags exist, or how they interact. Have one place where you manage all of your feature flags.

  1. Avoid ambiguously named flags

Give your flags easy to understand, intuitive names. Assume that someone other than you and your flag could potentially be using this flag days, months, and years into the future. Don’t have a name that could cause someone to turn it on when they mean off, or vice versa. For example “FilterUser”, when it’s off – does this mean users are filtered? or not?

  1. Have a plan for flag removal

Some flags are meant for permanent control, for example for an entitlements system. Other flags are temporary, meant for the purpose of a release only. If a flag can be removed (because it’s serving 100% or 0% of traffic), it should be removed, as quickly as possible. To enforce this rigor, when you write the flag, also write the pull request to remove it. That way, when it’s time to remove the flag, it’s a two second task.

22 Aug

Feature flagging and testing

Sam Stokes, Rapportive co-founder, has this tip on testing and feature flagging:

“I’ve found that explicitly testing both sides of the flag can help with managing the increased complexity a flag introduces.  It helps you catch changes down the line that would have broken the old, forgotten code path that 10% of customers are still on.  It provides reminders to clean up the old code and delete the flag (via occasionally breaking the tests).  And it provides some pushback against having too many interacting flags (if you had to write 8 tests for one function because it depended on three separate feature flags, maybe it’s time to refactor that function, or retire some flags).”

Sam Stokes regularly blogs here.

10 Aug

NDC Sydney: Developer Tools Down Under

I was honored to speak at NDC Sydney last week on “Faster and less risky releases with feature flags”. Just for clarification, NDC is “Norwegian Developer Conference”, which is so popular that they now have a London and Sydney event. NDC focuses on Microsoft technologies (.Net/Azure) and Agile. Attendees were a great crowd of Australians from Sydney, Brisbane, Perth and all over, as well as many international speakers. The talks were a good mix of practical (Damian Brady‘s demo on Octopus Deploy) to the theoretical (“What is Technical Debt?”). Adam Cogan had seen me speak at Microsoft Build and suggested I apply, and I was happy I attended.

At my talk on feature flagging, attendees were split between those who were already feature flagging and wanted tips on how to get better, and those who wanted to incorporate feature flagging into their development process. There was general buy-in that feature flagging was a valuable tool to help reduce risk and iterate quicker.

NDC Sydney did a good job of mixing in networking with talks. There was a Sydney Harbor Boat Cruise, as well as PubConf, with Ignite talks. The talks were so funny I wished I could have tweeted them, but organizer Todd Gardner said they were off the record.

I even managed to feature flag myself in Sydney. At a conference happy hour, I tell the group I’m here to speak about feature flagging:

IMG_8173 (1)

Sydney Flagged for Olympics

Guy: “Feature flagging? That’s the way all development is going.”
Me: “Yep! Absolutely”
Him “Do you think feature flagging as a service will take off?” he asks the group around us. Everyone jumps in with an opinion.
Him “There’s a company in Silicon Valley doing feature flagging as a service, can you believe it?”
Group weighs in with thoughts.
Then the guy pulls up a phone and shows everyone – the website of our company. LaunchDarkly.

At this point, it’s far more interesting to me to hear everyone else’s thoughts on feature flagging. It’s a loud bar at 7 pm, which is 2 AM my time. I’m jet lagged and soaking in candid feedback of the merits of feature flagging/toggling.

About 20 minutes later, I go talk to the original guy.

“Just so you know, I work for a feature flagging company…LaunchDarkly”.

Him: OH!

(more feature flagging geekery)

10 minutes later.. him – “So, the CEO of LaunchDarkly is a female.”

Me “Yes, that’s me”.

Him “!!!”

In short, I’d recommend NDC Sydney for any developer who wants both practical as well as cultural talks, as well as anyone who wants to connect with friendly Australian developers.

18 Jul

LaunchDarkly Turns Two!

Wow, two years goes by in the blink of an eye.

John Kodumal and I started LaunchDarkly with a firm belief that feature flags could help software product teams (product managers and engineers) iterate faster, with less risk. The beginning was exciting but hard. As founders of a two-person company you LaunchDarklyTurnsTwohave to decide the battle plan, then run around to the front to execute on the plan.  Feature flags were a relatively new technique arising from the convergence of continuous delivery, agile, DevOps and lean. Feature flags as a service was something completely new – the alternative was to build in-house, something not all teams could resource, or do without.

After two years, I’m thrilled that we’ve established “feature flags as a service” as a crucial piece of cloud infrastructure, with happy customers like CircleCI, Atlassian and AppDirect. Creating a category meant generating awareness and proving our value proposition. Our advisor Sean Byrnes (Flurry founder) said about his own days of mobile analytics “If you’re not out there in the water when the wave comes, it’s too late to get out there. But there’s a long time when you’re way out in the water.” Now, two years later, feature flags as a service is not so “out-there”. Our customers see that it helps them move faster with less risk.

Recently, Justin Darby, LaunchDarkly Sales Director, and I went to a new customer who’d just started using us. We had a forty developer kickoff, as we walked them through use cases and answered their questions about how best to use LaunchDarkly. At the end – the


LaunchDarkly Team

developers applauded, as they were so excited about how LaunchDarkly would help them iterate quicker ! I’ve been both a software vendor and a purchaser, and it is very, very seldom that a vendor gets applause. I’m incredibly gratified that we’ve assembled a team to build a product that is transforming software development.