18 Oct 2017

How We Beta Test at LaunchDarkly

Photo by Alex Holyoake on Unsplash

We recently looked at how some well-known companies beta test. Specifically we looked at groups that test in production, and do it well. As you know, testing in production is one of the best ways to find bugs and get solid feedback from your users. While some may shy away from this because of the risks involved, there are ways to mitigate risk and do it right. So this time we want to share how we beta at LaunchDarkly.

It’s no surprise that we dogfood at LaunchDarkly. Using feature flags within our development cycle is a straightforward process. We often push features directly into our production environment and safely test prior to allowing user access. When it’s time to beta test with users we can update the setting on the appropriate flag and get user feedback quickly. And of course, if we ever need to we can instantly turn features off.

Deciding Which Things to Build

When we’re thinking about new features to implement, we have our own ideas of which direction our product should go, but we also consider inbound requests. This can be from support tickets, questions from potential customers, or conversations with existing customers. Bottom line is we want to build a product that serves our customers, and so we do our best to listen to what they want.

Once we identify a feature we’d like to build—whether it was our own idea or a customer request—we’ll share it out to see if other customers are also interested. This is an important part of our beta testing process, because once the feature is in production and we’re ready to test it, these are the people we want to circle back with for beta testing.

Testing in Production

When it’s time to test, we test with actual end users in production. Our feature management platform allows us to turn features on for specific users. We can specify individual users, or we can expose users by attribute, like region (everyone in Denver)—and we can instantly turn them off at any time.

Because we’re testing in production, we don’t have to have an isolated environment or separate account. For those customers who showed interest, and agreed to participate in beta testing, we turn the features on in their production accounts.

Typically we beta for two weeks, sometimes as long as a month. As mentioned before, since we know which customers are interested in the feature, we can go back to them and have them test it. These are the users who already know they want this functionality, so we want to be sure it fits (or exceeds!) their expectations. And of course we want to make the most of this time, so it’s important we actually get feedback. We find that those who have asked for the feature are eager to let us know how things are working. We make a point of also following up with those who don’t proactively offer feedback—we want to hear from everyone!

While we’re testing and getting feedback, we’re taking all this information in and improving the feature before rolling it out to everyone else. When we feel confident we have something that’s ready to be shared, we’ll begin a percentage rollout to the rest of our users.

Embracing Failure

Using feature flags around features within our development cycles allows us to mitigate risk by pushing out small, incremental changes at a time. As you can see, this also enables us to beta test quickly and safely. If there are major bugs, we’re more likely to identify them early on before affecting our all of our customers.

“Embrace failure. Chaos and failure are your friends. The issue is not if you will fail, it is when you will fail, and whether you will notice.” -Charity Majors

Right now we’re currently in beta for scoped access tokens and a new faster .net SDK. Let us know if you’d like to take a look at it early, we’d love to hear what you think.

02 Oct 2017

Removing Risk from Product Launches: A webinar with LaunchDarkly, CircleCI, and GoPro

We recently sat down with one of GoPro’s Senior Engineering Managers, Andrew Maxwell, and the CTO of CircleCI to discuss reducing risk in product launches. Andrew talked about how his team delivered their code two weeks early, tested in production, and had an overall successful product launch. He goes into detail, sharing how his team uses continuous integration and feature flags to make product launches like that possible.

The Big Launch

Andrew’s team is responsible for web applications. Last September his team was focused on a big initiative around a product launch called GoPro Plus, which allows users to access and share content wherever they are. This launch included both mobile and desktop apps, and promoted two new cameras in the GoPro line.  

Two Weeks

The team delivered their code and pushed it into production two weeks early:

“…we used LaunchDarkly to push our code to production, turn the apps off — off by default—and then make sure that we had everything pushed out, deployed, and the infrastructure running live without customers actually seeing it.”

Two weeks gave them time to do a full integration test their features. They tested both in-house and in production—slowly opening up who had access—so they could get valuable feedback, find bugs, and make continuous improvements in the weeks leading up to the big launch. On the day-of, they were confident in their work and simply turned on 12 feature flags.

Check out the full webinar below to learn more about how the team used CircleCI and LaunchDarkly, their planning strategies, and best practices for their continuous integration pipeline.

26 Sep 2017

Do Away with Duct Tape: Infrastructure Rollouts a Safer Way

So I was told I need to write an introductory blog post for my first week at LaunchDarkly. Two months later and here I am writing my intro post. Seems like I got a little carried away writing code before getting to this! My name is Zuhaib and I will be working as Software Developer on anything back-end related. Previously I worked for a small startup called Atlassian on a chat product called HipChat.

Feature flags are not something new to me. I’ve seen a few homemade solutions in the past that always seem to leave me wanting more from them. Most systems would let you turn on or off features but had very limited ability to target users or control the percent rolled out. It was at Atlassian I got my first exposure to LaunchDarkly. We switched from our first in-house solution to LaunchDarkly and it was great. It helped us get new features out faster to our customers, and more importantly control features and backend service and disable them if they started to act up.

And as expected, we at LaunchDarkly do a lot of the same, using advance feature flags to make sure LaunchDarkly users have a good experience. One way we do that is by controlling rollout of infrastructure changes like database migrations.

Recently we needed to test an upgrade of our ElasticSearch cluster without impacting users. So we used a [percentage rollout] in LaunchDarkly and slowly targeted a subset of our users to the new cluster while we watched performance and stability.

Our flag allows us to control which cluster gets writes, which cluster gets reads, or which gets both. If we find a problem, we disable the flag and users go back to the other cluster. If it’s performing well we increase the percentage of users using the new database. You can see today we have rolled out the new cluster to 75% of users and working on getting it up to 100%. The code for this is as simple as adding a new statement that writes to either or both clusters:

While we may only be evaluating this second cluster for a short period of time, we’re actually leaving the flag in place, as it gives us a clean control mechanism for recovering from backups, or performing major version upgrades to ElasticSearch without customer impact.

This is just one of the many things you can do with LaunchDarkly flags for infrastructure changes.

08 Sep 2017

Saving Private Instances

You are a new Director of Engineering at an enterprise company. The company has just moved your product to the cloud and is hosting in Microsoft Azure. You come from a tech startup where you ran all of your software in the cloud and focused on building product. So naturally you have just implemented instrumented monitoring with HoneyComb, Kubernetes to manage your containers, and you’re thinking about leveraging serverless architecture.

You moved to this established enterprise company because they are providing technology to the healthcare industry and you are passionate about this space. They want you because you are good.

Now you face a challenge—how can I bring the good things from my past role and pair them with the security standards and compliance in my new role? Well, let’s first think about what the good things from your past role entails. Your previous team:

  • Deployed 5 times a day
  • Used feature branching
  • Feature flag first development
  • Implemented continuous integration

You realize you used 3rd party software for most of these good things, and had homegrown for others. You decide your first project is to research what software the new team is using, has built, and what software you will need to build and buy.

Security and the cloud.

Now, before we dive in, let’s consider the security standards and compliance you need to think about in your new role:

 

  • Where is my data stored?
  • What data is being sent over to the third party—is it PII?
  • Where connections are made to third parties?
  • How can I set different access control?
  • How it will work when connection fails-redundancy?
  • Is it HIPAA compliant?
  • How is all this audited?

 

Like your new company, many companies are just starting to move key operations to the cloud. This is SCARY. Moving into Azure will allow your company to free up floor space, add more server redundancy, easily scale up and down, only pay for what you use, and focus resources on revenue generating activities— security is still your responsibility.  And though these are all great things, you are not a security company.

Now you are tasked with bringing in these good processes, and they include 3rd party software providers. Products in this space are inherent in your development lifecycle and very close to your core application, but you need to ensure they are secure.

As we all know, most software providers in this space are cool tech companies in The Valley, far and isolated from the realities of your secure enterprise world. You look at on-premise options. And you remember that you’re driving change in the Healthcare space, not looking to manage infrastructure. The cloud options require a significant audit that is time consuming. A SaaS provider carries an ongoing risk that requires you store some data on 3rd party servers, which is a non-starter in Healthcare.

Is there a middle ground?

There is one more option many software providers offer: Private SaaS, also known as a managed instance, Dedicated VPC, or private instance. You surmise if they do not have on-prem, private SaaS, or a really really good security team, then it’s not likely their team is accustomed to working on enterprise challenges.

What exactly is a private SaaS offering? This refers to a dedicated single tenant, cloud software where the vendor manages the infrastructure.

Why choose a private instance?

  • Single Tenant—You will have a dedicated set of infrastructure contained within a VPC. This eliminates the risk of noisy neighbors.
  • Data Storage—Data can be stored in your AWS account or in an isolated section of the vendor’s AWS account. This allows flexibility, if you are in the EU, vendor could spin up the instance in AWS in the EU.
  • Residency—If you have a preference on where the instance is located or need to ensure proximity.
  • Compliance—If you have additional security or compliance regulations, a private instance can be customized to fit your needs.
  • Change Cadence—If you need to know when the software will be updated, you will have better insight and greater flexibility with a private instance.
  • Integrations—If you have custom tools, integrations can be built.

What’s next?

You have narrowed down a few software providers for each job you are trying to solve (continuous integration, branching, feature flagging). You understand the value, the costs, and the security implications. You have chosen to stay in the cloud, which makes your infrastructure team happy. You have chosen to use private instances and SaaS where it makes sense, which makes your security team happy. And you have the tools you need to bring the good things into your new role, so you are happy.

Now you can focus on helping your company deliver product faster, eliminate risk in your release process, speed up your product feedback loop, and do it all securely.

21 Aug 2017

How to Comply

Everyone loves a little affirmation (Image credit: twitter)

Things we wish we had known and things we were happy that we implemented early. 

At LaunchDarkly we recently embarked on the journey to SOC Type 2 compliance. While the reasons we chose to pursue this certification were primarily business driven, the tasks and actions incorporated into the process most directly impacted our engineering operations and development teams. And due to the experience and philosophy of the founding engineering team, the actually impact of the process was minimal.

Once deciding to sally forth on SOC certification (or any security and compliance certification for that matter) you will be in a much better position if you view the criteria of the certification as a benefit (or imperative) of good business practices. If you are in search of the certification primarily as a way to sell into certain customers or verticals, you will likely be frustrated with this entire process. Security is like diversity—you need to inherently believe in the value to be successful in implementations and outcome.

“So, what are these criteria you speak of?”

Need vs. want

The principle of least privilege is a well-known concept where you only provide access to the people that actually need it. This also extends to the level of access that is given. Ultimately this distills to: view vs control. I like to start here because this is a forcing function for so many of the other criteria. When you are thinking about who should have access to each system/service, you are inclined to:

  • Define roles
  • Log activity based on user/account
  • Build review process for accounts

On-boarding each new employee gives you an opportunity to review the tools you use to run your business, question who needs access, and to what extent.

This is also a great time to introduce new employees to another key security maxim: Trust, but verify. This concept is relevant in the context of the access that accounts or component services assert are needed, as well as the way that you validate user accounts.

For services—either 3rd party or just components of a larger service—you should know what information is being stored, and where. Ideally, you are being thoughtful of the first principle of least privilege.

This is especially true of customer data. The easiest way to protect data is to limit what you collect. Second is to make sure that you are being intentional and explicit with where you store the data. Finally, you need to look at the access policy that you put around that data.

For user accounts, multi-factor authentication (MFA) or two factor authentication (2FA) significantly increases your ability to validate that accounts are only being accessed by account owners.

Another time to plan for is employee off-boarding—ideally, before you off-board your first employee. This is also a good time to review your tools and access privileges.

Context; not blame

When a single developer is working on building a service, logs are primarily useful for understanding how well things are working (or where they broke). As the number of individuals working on the system becomes larger, the ability to know who changes what becomes increasingly important.

One caveat is that as you incorporate the ability to know who, you leverage this data to build context around why things changed, rather than simply using it as a means to place blame. If one person can bring down your service, then you probably should direct blame at the architect of the system. (Unless that destructive individual and the architect are the same person ¯\_(ツ)_/¯).

“A log without account context is like a novel without characters.”

A log without account context is like a novel without characters. You can build a picture of what happened, but likely will miss why things happened. If you don’t know why, you’re unlikely to prevent it from happening again.

Built for toddlers… or failure

Failure is trivial compared to proactive destruction (image credit: Lego City)

The stability of a service is often a strong indicator of the inherent security. After all, the most common exploitations are based on overloading some resource. Thinking further about the elimination of blame from building a secure and stable service, failures should be viewed as opportunities for increased robustness. This is where building for toddlers comes in.

Toddlers are the ultimate destroyers. It is the developmental stage where everyone starts to experimentally test the laws of physics. Gravity, entropy, projectile motion, harmonic oscillation—they’re all put through a battery of tests.

Ideally, you are thinking about your service from the perspective of a parent (or guardian) that is toddler-proofing their home. Bolt things down, put breakable things out of reach, lock up the flamethrower, and embrace the fact that you will miss things. For the items you miss, have an emergency procedure in place and appropriate medical supplies on hand.

Back in the context of your service…

Write it down… or it never happened

Your code/feature is not ‘done’ until the docs are written or updated. Services require constant supervision and are never ‘done’. But if a developer builds or changes something and doesn’t write it down, then they effectively become the only individual that is able to monitor or operate the entire service (at least with all the context).

So, what should you write down? “Everything,” is the easy answer, but often not the complete one. You want to write down enough to provide context if any component starts doing something unexpected.

If you don’t know where to start, a good approach is to write down what would need to happen if your service was deleted. How would you rebuild and restart your service? How would you restore your data?

Next you can look at the impact/process of the loss of each individual component service. The important part of this is to incorporate the documentation into the development process to ensure that as your service evolves your documentation is always up to date—otherwise, the change doesn’t exist after a failure.

Great, now you know what to do next time AWS S3 needs to reboot. But, what about your customers? The next step is to write down an action plan for service interruption. Make sure you have a process and plan in place for keeping folks in the loop.

Security is not the french fries

If you are in a situation where you are looking to “add” security, you are likely going to be in good company with Sisyphus. Security needs to be a part of your foundation—it is not an “add-on”. But if you are realizing this is now—that security is a requirement for your business— you can do more than wish you had considered it sooner. It is not too late, but it is not a quick fix that you can solve with a certification.

First you need to implement the security in your foundation and process. Make sure that it is part of your culture. Once you have a culture of security and process, compliance is just providing proof of your culture.

Now… about that certificate

You don’t show up to your official Genius Book of World Records judging day having never practiced juggling 9 clubs. Same goes for when you decide to get your certification for SOC.

However, when you build a strong foundation and culture for security and compliance, then the steps to get certification are rather straightforward. You call up your friendly neighborhood SOC auditor and get a copy of their check list.

In the case of LaunchDarkly we worked with the fine folks at A-lign. After an initial conversation we retained their services to conduct an independent audit of our systems and practices.

A few months prior to the audit, A-Lign provided our team with a checklist of all the documentation and proof they would need to see when they came onsite for our assessment. This afforded us the opportunity to ensure that all of our practices were organized and in a state that could be easily evaluated.

When the time came for the audit the auditor spent three days* on-site interviewing members of the team and reviewing our practices. After the on-site visit, we were informed that we had passed the initial competence certification.

Of course, now that we have gone through the validation process for one certification, it seems like a good time to keep going for a few more. Many of the certifications have a significant overlap in requirements. They are all looking to establish trust and ensure the service provider is operating in the best interest of the customer. And it turns out, most customers define trust in a very similar way.

18 Aug 2017

Risk Reduction and Harm Mitigation

Risk Reduction is trying to make sure bad things happen as rarely as possible. It is anti-lock brakes, vaccinations, clothing irons that turn off by themselves, and all sorts of things that we think of as safety modifications in our life. We are trying to build lives where bad things happen less often.

Harm Mitigation is what we do to make sure that when bad things do happen, they are less catastrophic. Fire sprinklers in buildings, seat belts, and needle exchanges are all about making the consequences of something bad less terrible.

What does that mean for a DevOps world where our risks and harms are very different? Charity Majors says we shouldn’t split the world into developers and operations, but into product and infrastructure—and I think that’s a useful way to think about risk too.

Product risk is problems that users experience. We can usually predict and mitigate the danger by testing and being aware of common failings. For example, we can expect and plan for users that may be on a flaky connection, or that they may try to exit a page without saving information. We can work around these problems because we know they’re out there. But our deployments are not something they should see until we’re ready for prime time. For this kind of risk, we use feature flags to control what content is delivered.

Infrastructure risk is more about the inherent fragility of delivering software. CDNs, fiber, servers, switches, towers…the whole system of getting data to people has failure zones. When we are trying to reduce infrastructure risk, we assume that latency is ever-present, networks are intermittent, and we won’t always be able to count on everything working right the first time. We try to build in robust and flexible ways than can route around failures. This is the place we might use feature flags to control failovers or to create circuit breakers to prevent flooding a fragile sector.

Product harm reduction is about making sure that users can have a positive experience, even if something happens that makes it less than ideal. We want to preserve their blog drafts, keep them from committing errors, only show them things they are allowed to change, and above all, avoid giving them a blank page. Something has gone wrong in the trip to the user, but they shouldn’t have to suffer for it.

Infrastructure harm reduction is the ability to pull back breaking fixes, shunt users away from vulnerabilities, and respond near-immediately to things that have gone very wrong. Harm reduction at this level is the kind of action a pager-responder can take to get things back on track before doing more intensive repairs and investigations in the morning.

In my first week, I spent a lot of time thinking about how to summarize our product for a variety of audiences. “Feature flags as a service” is short and pithy, but only works if you can bring your own definition of “feature flag” and a business understanding of why you would use them. What about “Feature flags allow you to make changes in near real-time instead of waiting for a deployment, and LaunchDarkly helps you manage and track flags across an organization”?

Well, that works, but it still doesn’t get to the core of why an organization would want to use feature flags. What I’ve come up with so far is this:

Feature flags segment the risk of creating a product into manageable parts.

Creating and deploying software is risky. We can accidentally build in errors, we can deliver it badly, or to the wrong people, and it can interact in unfortunate ways with existing software or hardware. As organizations, we want to do our best to do no harm and provide benefit. Using feature flags lets us wrap our features in decision points that we can then use to make life easier for our users.

Here are some types of risks that are reduced by using feature flags:

  • Server falls over from too much traffic
  • Canary launch is not well-tracked, problems are missed
  • Old features and workarounds are invisible and get left in place
  • Feature with vulnerable content is deployed
  • API endpoints are exposed to unauthorized users

Managing your feature flags is a post for another day—today I ask that you take a few minutes and think about how you can reduce risk and minimize harm in your organization, your project, or your code. How can you make things robust enough to resist failure, instrumented sufficiently to identify a failure spot, and flexible enough to reduce harmful consequences on the fly?