18 Aug 2017

Risk Reduction and Harm Mitigation

Risk Reduction is trying to make sure bad things happen as rarely as possible. It is anti-lock brakes, vaccinations, clothing irons that turn off by themselves, and all sorts of things that we think of as safety modifications in our life. We are trying to build lives where bad things happen less often.

Harm Mitigation is what we do to make sure that when bad things do happen, they are less catastrophic. Fire sprinklers in buildings, seat belts, and needle exchanges are all about making the consequences of something bad less terrible.

What does that mean for a DevOps world where our risks and harms are very different? Charity Majors says we shouldn’t split the world into developers and operations, but into product and infrastructure—and I think that’s a useful way to think about risk too.

Product risk is problems that users experience. We can usually predict and mitigate the danger by testing and being aware of common failings. For example, we can expect and plan for users that may be on a flaky connection, or that they may try to exit a page without saving information. We can work around these problems because we know they’re out there. But our deployments are not something they should see until we’re ready for prime time. For this kind of risk, we use feature flags to control what content is delivered.

Infrastructure risk is more about the inherent fragility of delivering software. CDNs, fiber, servers, switches, towers…the whole system of getting data to people has failure zones. When we are trying to reduce infrastructure risk, we assume that latency is ever-present, networks are intermittent, and we won’t always be able to count on everything working right the first time. We try to build in robust and flexible ways than can route around failures. This is the place we might use feature flags to control failovers or to create circuit breakers to prevent flooding a fragile sector.

Product harm reduction is about making sure that users can have a positive experience, even if something happens that makes it less than ideal. We want to preserve their blog drafts, keep them from committing errors, only show them things they are allowed to change, and above all, avoid giving them a blank page. Something has gone wrong in the trip to the user, but they shouldn’t have to suffer for it.

Infrastructure harm reduction is the ability to pull back breaking fixes, shunt users away from vulnerabilities, and respond near-immediately to things that have gone very wrong. Harm reduction at this level is the kind of action a pager-responder can take to get things back on track before doing more intensive repairs and investigations in the morning.

In my first week, I spent a lot of time thinking about how to summarize our product for a variety of audiences. “Feature flags as a service” is short and pithy, but only works if you can bring your own definition of “feature flag” and a business understanding of why you would use them. What about “Feature flags allow you to make changes in near real-time instead of waiting for a deployment, and LaunchDarkly helps you manage and track flags across an organization”?

Well, that works, but it still doesn’t get to the core of why an organization would want to use feature flags. What I’ve come up with so far is this:

Feature flags segment the risk of creating a product into manageable parts.

Creating and deploying software is risky. We can accidentally build in errors, we can deliver it badly, or to the wrong people, and it can interact in unfortunate ways with existing software or hardware. As organizations, we want to do our best to do no harm and provide benefit. Using feature flags lets us wrap our features in decision points that we can then use to make life easier for our users.

Here are some types of risks that are reduced by using feature flags:

  • Server falls over from too much traffic
  • Canary launch is not well-tracked, problems are missed
  • Old features and workarounds are invisible and get left in place
  • Feature with vulnerable content is deployed
  • API endpoints are exposed to unauthorized users

Managing your feature flags is a post for another day—today I ask that you take a few minutes and think about how you can reduce risk and minimize harm in your organization, your project, or your code. How can you make things robust enough to resist failure, instrumented sufficiently to identify a failure spot, and flexible enough to reduce harmful consequences on the fly?

17 Dec 2015

How LaunchDarkly’s Culture Inspires Its Product

Silicon Valley is known for its innovative work culture that cultivates employee happiness and promotes a harmonious work environment.  That’s all well and good, but does it translate into a great product?

I would argue that many companies promote an exciting ‘super duper awesome’ work environment, but have a hard time channeling that culture into a successful product.   Here, I’d like to discuss how we take this innovative culture and channel it into a successful product by promoting three core tenants.

  • Intellectual Curiosity – A culture of continuous intellectual growth allows everyone on the team to contribute to every aspect of the product, from marketing to design.  You are encouraged to learn, to adapt, and to challenge yourself and your teammates.  “What if’s” are encouraged and “dissent” is channeled through hypotheses and curiosity.  Product direction is shaped by the interplay of curiosity and discourse, even if you are not a domain expert in the subject at hand.
  • Reciprocated Support – Everyone on the team works for each other.  If I work on the weekend, it’s not because I have to or because I have a deadline, it’s because I want to.  Because I feel continuously supported, I reciprocate by trying to make everyone else’s workload easier and provide any benefit I can. We each have each others’ back, and that’s an inspiring feeling to have.
  • Vision Driven Development – While ultimately we are looking to be a successful business, we also truly believe that we are making things better and easier.  I find it very easy to pour my energy into something I truly believe in, rather than just something I feel will make money.

When you have a team that gels so well together, the product creates itself and reflects those values.  When each individual is held to a high standard, the product is held to a high standard.  Being proud of what you create is essential to a successful product.  We are proud of the work we do and we channel that pride to, not only make life better for our customers, but for each other.