08 Feb 2017

And so the adventure begins

It’s something of a LaunchDarkly tradition to write a blog post at the end of your first week. But here I sit, deep into week number two, and I haven’t written a thing. In my defense things move pretty fast around here, and I was just trying to keep my head above water. My second day on the job, having finally set everything up so I could start coding, they had me deploy production. By the time Friday rolled around, I had fixed 3 bugs, opened 6 pull requests, deployed production two more times — and written zero blog posts. Better late than never though, right?

A new chapter

I’m extremely thrilled to be part of this great company. I joined LaunchDarkly after nearly 10 years at Atlassian, where I spent most of my time doing front-end development and arguing about UX. I’m excited to take everything I learned there and continue to make LaunchDarkly a product people love to use.

I took the job because I believed in the people, the product, and the culture. If anything I underestimated all of those things and I can’t wait to see where this journey takes me.

An escape

The other day we went on a team outing to an escape room. If you’re not familiar, you basically solve a series of puzzles using items in a room in order to “escape” the room. I’ve never done one before, so I have nothing to compare it to, but this particular escape room required a lot of coordinated effort. We had 12 people participating (almost the whole team, which is at 15 now), but at any given time there were probably 3-5 puzzles being worked on.

The beginning was a little chaotic as everyone tried to figure out exactly what we’d gotten ourselves into and what in the world to do next. But pretty soon we settled down, started self-organizing, and talking through the problems. It was pretty amazing to see everyone working together so well — no arguments or drama, no jockeying for leadership, just good old-fashioned teamwork. I think that experience sums up my time at LaunchDarkly so far — fast and fun, with a great group of people working together towards a common goal.

Oh yeah, and we escaped the room (just barely).

04 Jan 2017

How Feature Flagging Helps Usability Tests

Usability testing in a real-world environment (aka production environment) gives us insight into how users actually use our product in their day to day lives. It is one thing to run a test in a lab setting, but it is another to have users try features while they are walking, running to the airport, stressed, and sleepy.

But, no matter how hard we try, it is very difficult to truly simulate how our apps behave in our production environment. We can run focus groups, beta tests on a beta environment, and test things internally, but how can we truly mimic the real world in an artificial context? In other words, how do we test a feature while also simulating the environment it’s meant to be used in?

LaunchDarkly Usability Testing In Production - Feature Flags and Feature Toggles - Context

A good real-world usability test analyzes features efficiently and accurately collects user feedback to improve the user experience. But, when we test anything in a non-production environment, we are inherently biasing our tests. In a lab-based usability test:

  1. Users are overly cognizant of the feature they are testing.
  2. Users are unnaturally focused on testing that new feature.
  3. Users are using fake data or incomplete data, and don’t sufficiently utilize actual use cases.
  4. It is very hard to test discoverability (i.e can a user find the feature on their own?).
  5. Users tend to ignore distractions, like external notifications (Facebook, Skype, texts) and just focus on the task at hand.
  6. Users are in an overly analytical mode, typically looking to give feedback.
  7. Users are overly forgiving, looking to please!

Does this mean that running usability tests in non-production environments is a waste of time? Not at all! This is absolutely necessary to identify bugs, check for general usability, and solicit feedback quickly.

However, it should just be one of the steps in a comprehensive usability testing process, one that involves internal, staging, and production usability testing.

Benefits of usability testing in production:

The primary purpose of usability testing in production is to gather real-world user behavior while minimizing bias and performance risk. Some more benefits include:

  1. Genuine user feedback in a real-world environment
    • Quantitative insight into your feature’s performance. How well is it scaling? Are users using it? How is it impacting your system? How are your levels of engagement?
    • Contextual insight into your feature’s efficacy. Do users see the new feature? How is it meshing within the context on your existing feature set? Are people using it as intended? Are people using it once and then not using it again?
    • Qualitative feedback. Are users complaining? Are they happy? Are they neutral?
  2. No opt-in bias – users test the feature without knowing they are part of the test. You can assess how well they use it by using a product like Full Story to record the session or by tracking metrics. You therefore get a more representative sample testing the feature, rather than just early adopters.
  3. Measuring actual system performance – there is nothing quite like your production environment, where you can have a complex array of nodes, clusters, CDNs, etc allowing your app to scale. As people start to use the new feature, you can see how it is impacting your actual system (load times, discoverability, caching issues).

Managing a usability test in production:

Of course, testing anything in your production environment is inherently risky and has real-world consequences. If you launch something to everyone just to get feedback and they hate it, then you risk permanently losing those users. Equally bad, you can cripple your entire application with unforeseen scaling and performance issues.

To mitigate this, companies like Facebook, Amazon, and Google collect production feedback by releasing features behind feature flags. While we won’t go into to the specific anatomy of a feature flag, we can go through the methodology behind the release.

LaunchDarkly - Usability Testing Using Feature Flags and Toggles - Betas and Feedback

If a feature is wrapped in a feature flag, then it gives you control over who sees the feature and when. This means that you can perform targeted and controlled releases using a percentage rollout, whereby you can incrementally increase a feature’s visibility to 1%, 5%, and to 100% of your users.

Hence, you can collect production level feedback because you control the level of risk. If a new feature is performing well, then you can keep increasing the percentage rollout. If it is tanking or hurting performance, you can reduce the rollout or kill it completely.

Therefore, feature flags (aka feature toggles) give you full control over the risk of your production releases. You can gather real-world user feedback by separating your feature rollout from your code deployment.