23 Mar 2018

The First Week of a Wild Ride

My first week on the job, we raised series B funding and celebrated with a trip to a local tiki bar. After that big milestone, I’ve split my time learning a new programming language, learning a new deployment toolchain, getting acquainted with all my great new colleagues, and realizing that there’s so much more for me to learn. A few things, however, have been familiar.

At my last job, we used LaunchDarkly for feature management. Engineers would create feature flags and then ship features behind those flags. We kept those features hidden to customers until we had tested them out internally. Once a feature was ready, we’d let our customer success team communicate the change to our customer and enable the feature.

Something that brings me pangs of nostalgia is operational issues with recording event data. At my last gig, we had an outage because of a bug (I introduced) that created a channel in RabbitMQ every time that we published an event. This bug went undetected for a while. Then, one day we had a load spike that caused our RabbitMQ server to run out of memory and prevent us from accepting all writes (because we tried to publish before the transaction was committed). Since the bug went undetected for so long, reverting recent deploys didn’t have an effect, and so we had an outage until we were able to find the root cause of the issue. Once we knew the root cause, the fix was easy, but while we were investigating, our application was essentially read only. If we had the ability to disable publishing for the one customer that was sending us the spike in traffic, we’d have been able to isolate and reduce the severity of this outage.

At LaunchDarkly, luckily, we use feature flags for those operational scenarios. Just recently we were getting a dramatic spike of events from one customer, to the point it was causing increased latency in our message broker. Once we noticed that, we disabled publishing events for that customer since they were already over their usage limit. By doing that we were able to isolated degraded service and prevent issues cascading to anyone else. Once we flipped the feature flag in LaunchDarkly, in real-time, the traffic was diverted and the outage was averted.

Building resilient systems like this is something that I’m really excited to be doing. I’ve got a lot to learn and the company is growing fast, so I think this wild ride is gonna continue for a while.

19 Mar 2018

Go Serverless, not Flagless: Implementing Feature Flags in Serverless Environments

A demonstration of LaunchDarkly feature flags with AWS Lambda

Serverless architecture, while a relatively new pattern in the software world, is quickly becoming a valuable asset in a variety of use cases.

There are some notable business advantages to serverless architecture:

  • Faster development – easily harness existing backend services and third party APIs to quickly build and deploy apps
  • Lower cost – substantially decreased operational costs by not having to manage infrastructure in addition to the componentization of the entire application (only scale what you need to scale)
  • Less maintenance – scaling, load balancing, and patches are a thing of the past, serverless platforms take care of it all for us

Coupled with feature flags, a serverless application allows teams to control feature releases without having to redeploy. This is especially useful for serverless applications that tend not to focus on persistence and be more stateless in nature.

In this article, we’ll start with the assertion that you have already decided that serverless functions makes the most sense for your application architecture. What you really need is the ability to continue using feature flags in this type of architecture. This is where LaunchDarkly has got you covered.

We’re going to look at how to integrate LaunchDarkly into an application running on AWS Lambda. AWS Lambda lets you run code without provisioning or managing servers. On the other hand, LaunchDarkly is a feature flag management platform that allows you to easily create and manage feature flags at scale.

For smaller workloads, you can use the LaunchDarkly SDK as you normally would. Evaluating a feature flag with the nodejs runtime in AWS Lambda looks like this:

This naïve approach is appropriate for serverless workloads that have limited concurrency (e.g., a scheduled job). If you have a workload that has high levels of concurrency or sensitivity to cold starts, this approach will have poor cold start performance because we need to call out to LaunchDarkly whenever a new Execution Context is created. To overcome this, we need to add a second Lambda function, an API Gateway, and ElastiCache to build a serverless flag storage pipeline. We’ll use this pipeline to maintain a store of feature flag configurations so that the application can just get flags from redis and avoid a more costly cold start.

Storing Flag Configurations

AWS turns out to be a well-equipped platform for serverless feature flagging. In our implementation, we create a webhook in LaunchDarkly to hit an API Gateway endpoint which acts as a trigger for invoking a lambda function.

Then, the lambda function can have LaunchDarkly populate a Redis ElastiCache cluster by initializing its SDK with the built-in “RedisFeatureStore” option. This lambda function built on the NodeJS runtime would look something like this:

Note that we’re initializing the LaunchDarkly SDK inside the handler. That’s because, by initializing the SDK, we’ll pass the update that triggered the webhook along to the redis store. Generally speaking

Processing Feature Flags

Now that our flag configurations are neatly stored in the Elasticache cluster, we need to change our naïve Lambda function to retrieve and process flags without waiting for any outbound connections to LD! This is achieved by enabling LaunchDarkly Relay mode in the SDK options. The function’s “event” parameter can be used to pass in a user to be evaluated by the LaunchDarkly SDK.

Take Aways

It’s important to understand that there are multiple ways to implement feature flags in your serverless architecture. The method will depend on your app’s structure, your serverless provider, and your implementation timeline. You can check out this guide to feature flagging for best practices and ways to get started.

Originally published on April 14, 2017 by Justin Baker. Updated on March 19, 2018 by Mike Atkins.