04 Jan

Running Usability Tests in Production

Usability testing in a real-world environment (aka production environment) gives us insight into how users actually use our product in their day to day lives. It is one thing to run a test in a lab setting, but it is another to have users try features while they are walking, running to the airport, stressed, and sleepy.

But, no matter how hard we try, it is very difficult to truly simulate how our apps behave in our production environment. We can run focus groups, beta tests on a beta environment, and test things internally, but how can we truly mimic the real world in an artificial context? In other words, how do we test a feature while also simulating the environment it’s meant to be used in?

LaunchDarkly Usability Testing In Production - Feature Flags and Feature Toggles - Context

A good real-world usability test analyzes features efficiently and accurately collects user feedback to improve the user experience. But, when we test anything in a non-production environment, we are inherently biasing our tests. In a lab-based usability test:

  1. Users are overly cognizant of the feature they are testing.
  2. Users are unnaturally focused on testing that new feature.
  3. Users are using fake data or incomplete data, and don’t sufficiently utilize actual use cases.
  4. It is very hard to test discoverability (i.e can a user find the feature on their own?).
  5. Users tend to ignore distractions, like external notifications (Facebook, Skype, texts) and just focus on the task at hand.
  6. Users are in an overly analytical mode, typically looking to give feedback.
  7. Users are overly forgiving, looking to please!

Does this mean that running usability tests in non-production environments is a waste of time? Not at all! This is absolutely necessary to identify bugs, check for general usability, and solicit feedback quickly.

However, it should just be one of the steps in a comprehensive usability testing process, one that involves internal, staging, and production usability testing.

Benefits of usability testing in production:

The primary purpose of usability testing in production is to gather real-world user behavior while minimizing bias and performance risk. Some more benefits include:

  1. Genuine user feedback in a real-world environment
    • Quantitative insight into your feature’s performance. How well is it scaling? Are users using it? How is it impacting your system? How are your levels of engagement?
    • Contextual insight into your feature’s efficacy. Do users see the new feature? How is it meshing within the context on your existing feature set? Are people using it as intended? Are people using it once and then not using it again?
    • Qualitative feedback. Are users complaining? Are they happy? Are they neutral?
  2. No opt-in bias – users test the feature without knowing they are part of the test. You can assess how well they use it by using a product like Full Story to record the session or by tracking metrics. You therefore get a more representative sample testing the feature, rather than just early adopters.
  3. Measuring actual system performance – there is nothing quite like your production environment, where you can have a complex array of nodes, clusters, CDNs, etc allowing your app to scale. As people start to use the new feature, you can see how it is impacting your actual system (load times, discoverability, caching issues).

Managing a usability test in production:

Of course, testing anything in your production environment is inherently risky and has real-world consequences. If you launch something to everyone just to get feedback and they hate it, then you risk permanently losing those users. Equally bad, you can cripple your entire application with unforeseen scaling and performance issues.

To mitigate this, companies like Facebook, Amazon, and Google collect production feedback by releasing features behind feature flags. While we won’t go into to the specific anatomy of a feature flag, we can go through the methodology behind the release.

LaunchDarkly - Usability Testing Using Feature Flags and Toggles - Betas and Feedback

If a feature is wrapped in a feature flag, then it gives you control over who sees the feature and when. This means that you can perform targeted and controlled releases using a percentage rollout, whereby you can incrementally increase a feature’s visibility to 1%, 5%, and to 100% of your users.

Hence, you can collect production level feedback because you control the level of risk. If a new feature is performing well, then you can keep increasing the percentage rollout. If it is tanking or hurting performance, you can reduce the rollout or kill it completely.

Therefore, feature flags (aka feature toggles) give you full control over the risk of your production releases. You can gather real-world user feedback by separating your feature rollout from your code deployment.

14 Oct

The Future of Feature Flags: Managing Dynamic Applications

LaunchDarkly Multivariate Feature Flags / Toggles

Traditionally, software teams have used feature flags/toggles to control simple rollouts and enable kill switches.  Boolean values were used for “on” or “off”, “true” or “false”.  With the introduction of multivariate flags, developers have been experimenting with serving rich values via these flags: strings, numbers, JSON objects, and JSON arrays. This has opened up a whole new world of dynamic application management.

Use Cases

Using feature flags to manage dynamic applications opens up many powerful and interesting use cases, for example:

  • Manage features in a pricing plan
  • Customize pricing based on user segment Apply coupons and discounts based on user actions or holiday sales
  • Allow users to personalize their own experiences
  • Manage CSS styling to test colors, layouts, and elements
  • Control conditional logic separately from your code base

The Multivariate Flag

A multivariate feature flag has two primary benefits: you can customize the number and type of variations returned.

LaunchDarkly Multivariate Feature Flags / Toggles

Let’s first look at a simple example. This feature flag calls the variation method to determine which variation the user should get for a “checkout.flow” feature flag.

Here, the user “bob@example.com” will either see a one-click, two-click, or original checkout based on the string value returned by the feature flag:

Serving Dynamic Values

Now, let’s use a feature flag to insert prices into a pricing page based on some attributes of a the user. We’ll call this feature flag “plan.price”.

LaunchDarkly Managing Dynamic Content with Feature Flags / Toggles

Here, the user will receive a price of $20, $50, or $75 depending on whether they match a particular targeting rule.

For a more advanced use case, these values (20, 50, 75) could be used to populate other pricing dependencies, like a customer invoice and account limits.

Best Practices

Of course, you should not use feature flags as a secondary database, but they could be useful as a way to personalize feature targeting without tying that logic into your codebase. This allows you to rapidly change pricing models, test color schemes, and configure complex features without having to redeploy.

07 Sep

Why microservices need feature flags

Microservices is the practice of breaking up a huge, monolithic release where all components are tested and released as a whole, into many discrete services that can go on independent release schedules. The benefits of microservices were popularized by Martin Fowler, and put into practice by Amazon and Netflix. However, with microservices, releases become more complicated, by n-factorial. Instead of one monolith that can tested as a whole, if there are even as few as nine different services, each needs to be tested with every other component. Suddenly there are nine potential friction points.

Feature flags to the rescue! With feature flags, engineering teams can have complete control over their various microservices. First, wrap the microservice with a feature flag, with all traffic going to the old version. Then, release a new version. With a feature flag, gradually put whatever traffic you want to the new version, verifying functionality in all the other micro services. The new traffic and be cordoned off however suits your business. A feature flag at it’s simplest can be an “on” “off” switch to quickly flip between versions, similar to a blue green deployment. However, feature flags can serve arbitrarily complex (or simple) variations of traffic to the new microservice. Some example of how to test the new micro service are
– percentage rollout
– by IP address of service
– by version number of other services
or whatever other information. Feature flags allow users to do “microdeployments” of microservices. A microdeployment is breaking a deployment into smaller components of who exactly receives the new service.

Using feature flags and microservices together allows engineering teams to truly unlock the value of decoupling. You can read more about how we use microservices and feature flags at LaunchDarkly here.

 

LAUNCHDARKLY HELPS YOU BUILD BETTER SOFTWARE FASTER BY HELPING MANAGE FEATURE FLAGS AT SCALE. START YOUR FREE TRIAL NOW.

 

26 Aug

3 Ways to Avoid Technical Debt when Feature Flagging

Feature flags are a valuable technique of separating out release (deployment) from visibility. Feature flags allow a software organization to turn features on and off at a high level, as well as segment out their base to allow different users different levels of access. However, feature flags have an (ill-deserved) reputation of “Technical Debt”. Used incorrectly, feature flags can accumulate, add complexity, and even break your system. Used correctly, feature flags can help you move faster. Here’s three easy ways you can avoid technical debt when using feature flags.

  1. Create a central repository for feature flags

Using config files for feature flags is “the junk drawer” of technical debt. If you have seven config files with different flags for different parts of the system, it’s hard to know what flags exist, or how they interact. Have one place where you manage all of your feature flags.

  1. Avoid ambiguously named flags

Give your flags easy to understand, intuitive names. Assume that someone other than you and your flag could potentially be using this flag days, months, and years into the future. Don’t have a name that could cause someone to turn it on when they mean off, or vice versa. For example “FilterUser”, when it’s off – does this mean users are filtered? or not?

  1. Have a plan for flag removal

Some flags are meant for permanent control, for example for an entitlements system. Other flags are temporary, meant for the purpose of a release only. If a flag can be removed (because it’s serving 100% or 0% of traffic), it should be removed, as quickly as possible. To enforce this rigor, when you write the flag, also write the pull request to remove it. That way, when it’s time to remove the flag, it’s a two second task.

04 Dec

Why smart companies choose LaunchDarkly for feature flags as a service

Feature flags are a key part of successful continuous delivery that the most successful companies like Heroku and CircleCI use to supercharge their development. Feature flags power software teams to release features faster to the users they want. The biggest companies maintain their own feature flagging framework – but the smartest ones are using feature flags as a service, from LaunchDarkly.

Feature flags start off simple, but can quickly become a quagmire when you need to maintain them. At the beginning, it’s easy enough to create a feature flag as a database table that an engineer can manually use to turn a user on or off. Next, you’ll want to start doing more sophisticated rules like “20% of traffic,” “everyone on our beta email list” or “no one with a TechCrunch email address.” At this point, usually a rough UI is made so that non-technical people can also control. Then, the bright idea comes of using the same framework for a/b testing. This works, until it doesn’t. That’s when the smart companies switch to LaunchDarkly for their feature flags support, for ROI, reuse, maintenance and sophistication.

ROI Auction.com chose LaunchDarkly because they didn’t want to dedicate multiple engineers to build a feature flagging system from scratch. They wanted their engineers to work on differentiating features for Auction.com. Lawrence Yuan, Senior Director Web and Mobile Engineering at Auction.com, LLC says “LaunchDarkly has been easy to on-board and integrate into our existing development cycle, which has helped us move faster with less risk.”

Reuse One large LaunchDarkly customer had built a custom feature flagging structure for their one product. When they acquired another company, they found that they couldn’t reuse the functionality for any acquisitions, as it had been intertwined with their own functionality. Another customer recently rebuilt from Ruby to Node.js. When they rebuilt, their existing feature flagging infrastructure was obsolete. In both cases they found it prudent to move to a more stable system.

Maintenance costs CircleCI had built and maintained their own feature flagging system, as it was a critical part of the release process they used for canary releases and dark launch. However, they wanted to spend their time on continuous integration, not on their feature flagging system. Mark Pundsack, VP Product CircleCI, made the decision to move to LaunchDarkly, saying “I really like LaunchDarkly, and I’m glad we’re using it. I believe that feature flags is a critical piece of modern development, and LaunchDarkly brings this part of continuous delivery to the masses.”

Flexibility & sophistication LaunchDarkly facilitates both simple and complex feature flagging. We have both per feature views as well as per user views, so you can easily see who has what features. We allow you to see how long a feature flag is in use so you know when to remove it. Other developer tools companies use LaunchDarkly, saying “We like that you think about nothing but feature flags.”

The smartest companies are choosing to have LaunchDarkly handle their feature flagging. Are you ready to join them? Click here to start your free trial.

30 Sep

The Design Void of the Developer World

The challenge of effectively designing for continuous delivery.

Developers are expected to use increasingly powerful tools to deliver increasingly complex and innovative software. So, why do developer tools often feel like an IRS tax form?

I argue that it stems from a culture that treats aesthetics and functionality as exclusive, whereas I would argue they are highly intertwined. When software is consumer facing, we see a sweeping trend of minimalist elegance aimed at cultivating an emotional connection with the user. But, when that target user is now a developer, design seems to take a back seat to functionality.

“It just needs to work! Not look good!” is what I’ve heard in the past. This is the notion that accurately performing a task correlates to function; that putting in the correct inputs and receiving a reasonable output is the benchmark of success.

But shouldn’t there be other indicators of success? What about time saved? What about ease of use? What about scalability and reliability? What about the ability to make informed decisions? I can show someone a spreadsheet of tabular data or I could show them some informative charts. Both present the same data. Both functionally deliver. But one works much better at delivering the information and fulfilling its purpose.

Let’s look at continuous delivery: the art of releasing better software in shorter cycles. It’s the amalgam of time management, planning, and iterating. It would make sense that developing a product for continuous delivery would be cognizant of this interplay.

At LaunchDarkly, we could have taken two approaches with our continuous delivery tool.

  1. Functional Model: “Let’s just make the product work.”
  2. Empowerment Model: “Let’s make sure our product saves time, mitigates aggravation, and allows for better decision making.”

Let’s assume that both models have equivalent functionality and backend logic. Truly harnessing the entire functional suite requires intuitive access. This is where design comes in. Design is about access and empowerment. It is about empowering the user through access to features, knowledge, and informed decision-making.

Developer tools do not have to be barren and sparse. They can be fun, interactive, and powerful without sacrificing efficiency. While designing for developers is different than designing for consumers, those differences manifest in information delivery and should not sacrifice empowerment.