10 Jan

My first week at LaunchDarkly was surreal.

The Space
LaunchDarkly’s office space is located inside Heavybit. The building has an industrial theme and feel with metal and wood everywhere. I’m surrounded by three walls of windows that provides natural light on an open floor plan. So warm and beautiful…but that’s just the start. Heavybit provides many interesting amenities such as meetup space, workshops, not to mention a kitchen full of drinks and snacks, and lunch served everyday. The service has been amazing, keep up the great work Heavybit!

Day One?
I joined on the company’s first day of the new year. Everyone’s back from the holidays, telling stories, exchanging ideas, working, having lunch together and going to a few meetings. The team and leadership were very welcoming and helpful to get me up and running. It felt like a “low key” day compared to a day at my past startups. However, as the days go on I realized that this kind of day is EVERY day at LaunchDarkly. Every day leadership is there to guide and support the team, and vice versa. Everyone is at arm’s reach and easy to talk to. Which I love because it keeps meetings to a minimum and allows autonomy to GSD. Working with this intelligent, colorful and compassionate group has exceeded my expectations of what startup culture can be.

I am One with the Force…
At the end of the week, I felt I was contributing to a great team building an awesome product that is helping companies produce more awesome products and services. As a business operations person, this is music to my ears – be a force multiplier!

In weeks to come, I look forward to taking on challenges and growing the team. I’m excited to learn from and contribute to the LaunchDarkly community.

04 Jan

Running Usability Tests in Production

Usability testing in a real-world environment (aka production environment) gives us insight into how users actually use our product in their day to day lives. It is one thing to run a test in a lab setting, but it is another to have users try features while they are walking, running to the airport, stressed, and sleepy.

But, no matter how hard we try, it is very difficult to truly simulate how our apps behave in our production environment. We can run focus groups, beta tests on a beta environment, and test things internally, but how can we truly mimic the real world in an artificial context? In other words, how do we test a feature while also simulating the environment it’s meant to be used in?

LaunchDarkly Usability Testing In Production - Feature Flags and Feature Toggles - Context

A good real-world usability test analyzes features efficiently and accurately collects user feedback to improve the user experience. But, when we test anything in a non-production environment, we are inherently biasing our tests. In a lab-based usability test:

  1. Users are overly cognizant of the feature they are testing.
  2. Users are unnaturally focused on testing that new feature.
  3. Users are using fake data or incomplete data, and don’t sufficiently utilize actual use cases.
  4. It is very hard to test discoverability (i.e can a user find the feature on their own?).
  5. Users tend to ignore distractions, like external notifications (Facebook, Skype, texts) and just focus on the task at hand.
  6. Users are in an overly analytical mode, typically looking to give feedback.
  7. Users are overly forgiving, looking to please!

Does this mean that running usability tests in non-production environments is a waste of time? Not at all! This is absolutely necessary to identify bugs, check for general usability, and solicit feedback quickly.

However, it should just be one of the steps in a comprehensive usability testing process, one that involves internal, staging, and production usability testing.

Benefits of usability testing in production:

The primary purpose of usability testing in production is to gather real-world user behavior while minimizing bias and performance risk. Some more benefits include:

  1. Genuine user feedback in a real-world environment
    • Quantitative insight into your feature’s performance. How well is it scaling? Are users using it? How is it impacting your system? How are your levels of engagement?
    • Contextual insight into your feature’s efficacy. Do users see the new feature? How is it meshing within the context on your existing feature set? Are people using it as intended? Are people using it once and then not using it again?
    • Qualitative feedback. Are users complaining? Are they happy? Are they neutral?
  2. No opt-in bias – users test the feature without knowing they are part of the test. You can assess how well they use it by using a product like Full Story to record the session or by tracking metrics. You therefore get a more representative sample testing the feature, rather than just early adopters.
  3. Measuring actual system performance – there is nothing quite like your production environment, where you can have a complex array of nodes, clusters, CDNs, etc allowing your app to scale. As people start to use the new feature, you can see how it is impacting your actual system (load times, discoverability, caching issues).

Managing a usability test in production:

Of course, testing anything in your production environment is inherently risky and has real-world consequences. If you launch something to everyone just to get feedback and they hate it, then you risk permanently losing those users. Equally bad, you can cripple your entire application with unforeseen scaling and performance issues.

To mitigate this, companies like Facebook, Amazon, and Google collect production feedback by releasing features behind feature flags. While we won’t go into to the specific anatomy of a feature flag, we can go through the methodology behind the release.

LaunchDarkly - Usability Testing Using Feature Flags and Toggles - Betas and Feedback

If a feature is wrapped in a feature flag, then it gives you control over who sees the feature and when. This means that you can perform targeted and controlled releases using a percentage rollout, whereby you can incrementally increase a feature’s visibility to 1%, 5%, and to 100% of your users.

Hence, you can collect production level feedback because you control the level of risk. If a new feature is performing well, then you can keep increasing the percentage rollout. If it is tanking or hurting performance, you can reduce the rollout or kill it completely.

Therefore, feature flags (aka feature toggles) give you full control over the risk of your production releases. You can gather real-world user feedback by separating your feature rollout from your code deployment.

22 Dec

To Be Continuous: Goal Setting

In this episode, Edith and Paul talk about OKRs and other goal-setting methods. Hear them discuss the good, the bad and the “gameability” of measuring success. The pair remind us that there are learnings in the journey, not just the destination, despite what the term “growth hacking” might have you believe. This is episode #28 in the To Be Continuous podcast series all about continuous delivery and software development.

Continue reading “To Be Continuous: Goal Setting” »

22 Dec

A New Way To Beta Test

LaunchDarkly Feature Toggle Beta

It’s best practice for products to have some sort of beta – a way to collect customer feedback and test performance before releasing to everyone. In an era of continuous delivery, we are delivering new features and experiences more frequently and with less time to gather thorough customer and performance feedback. With this increased cadence, product teams are having to make betas shorter, forego them altogether, or slow down their release cadence to gather adequate customer feedback.

Challenges of traditional betas:

  • Coordinating Opt-Ins: It sometimes takes weeks or months to gather customer opt-ins to test new betas. You also have to organize the distribution of beta keys (ex. for early access to games) and reminder emails.
  • Organizing Focus Groups: Getting feedback from focus groups is often time consuming and expensive, creating a long feedback loop that lengthens the release process.
  • Opt-Out: If customers opt-in to a beta and don’t like the experience, then they will want a simple mechanism to switch back to the production version.
  • Granular Betas: It is very difficult to do targeted betas based on user attributes or to perform incremental percentage rollouts of new beta features.

Feature toggles

To overcome these challenges, smart product teams are beginning to run betas with feature flags/toggles. These are mechanisms for granularly controlling software releases, allowing you to control the timing and visibility of a beta release.

Currently, many betas are tied to code releases and are managed by a config file or database.  This approach requires engineering time or custom mechanisms to opt-in users.

With feature toggles, you can empower product, marketing, sales, and even customers (themselves) to opt-in new to a new beta experience.

Feature Toggle Beta Test LaunchDarkly

In this simple example, you can use a toggle to control the visibility of a new beta feature. Ideally, this toggle would be part of a user interface that could be controlled by a non-technical team member. The code, itself, could be deployed off and then turned on via the toggle.

Beta Test Percentage Rollout with Feature Toggle LaunchDarkly

Moreover, you can also use the toggle to control the percentage of users who get the beta experience. For instance, you could release the new beta experience and have it rolled out to 0% of users. You could gradually increase the rollout percentage from 1% to 5% to 20% and more, collecting customer and performance feedback along the way.

Surfacing this beta control functionality in a user interface is critical for giving non-technical team members access to release controls.

Regional betas

For a recent example of a targeted rollout, we can look at how Pokémon GO released their product country by country: first to the United States and then abroad.

This is a great use case for feature toggles because you can create targeting rules to determine which users receive the feature first. For example, I could create a toggle that is governed by the rule: “If users live in San Francisco, then serve the new Nearby Pokémon feature”. This allows you to maintain different regional feature sets without having to deploy different versions of the application. It also allowed Pokémon GO to refine their algorithms and assess customer feedback before rolling out the new feature to a wider audience.

Benefits of beta testing with feature toggles

  • Empowered non-technical users: Allow the sales, marketing, product, design, and business teams to turn features on for specific users, collect feedback, and control the business logic. This also substantially cuts down on engineering time.
  • Production feedback for your beta tests: Test features in production with limited user segments to collect customer and performance feedback.
  • Incremental percentage rollouts: Gradually roll out features to incrementally test performance and mitigate risk. If the feature is bad, toggle it off.
  • Real-time opt-in / opt-out: Allow users to opt in and out of beta tests in real time, controlled via the feature toggle. Skylight provides a nice article on this.

Getting started with toggles

Conceptually, a feature toggle is relatively simple. You create a conditional in your code that controls the visibility of a code snippet. There are many open source libraries that will allow you to get started.  However, these libraries become cumbersome when you try to feature toggle at scale or restrict access to particular toggles. Depending on your needs, you could consider a feature toggle management platform to provide a system for access control and mitigating technical debt.

20 Dec

LaunchDarkly raises $8.7 million for feature flag management: Separating business logic from code

I’m pleased to announce that we’ve raised an $8.7M Series A  with Josh Stein of DFJ leading the investment and joining our board. John (cofounder and LaunchDarkly CTO) and I are thrilled about the next chapter for LaunchDarkly, the leading feature flag management platform.

Our A funding is a marker to reflect on our progress since we announced our seed round in June 2015. Then, we were a team of four with our product in private beta. Now, we serve billions of feature flags (daily) to amazing customers like AppDirect, CircleCI, Lanetix and Upserve. Microsoft recommended LaunchDarkly for feature flagging, and Atlassian showcases how we help them do DevOps right. As a former Product Director at TripIt, I initially thought our target market would be mobile app developers. I was right – mobile app developers do love us for how we can help them iterate faster. However, the market is so much bigger – LaunchDarkly is feature flag management for literally anyone who codes. ANYONE. Whether that team is reducing riskmanaging subscription plans or  microservices – we can help them. Part of our funding will go towards hiring more amazing engineers to continue to build feature flag management. We’re hiring!

We believe firmly that using feature flags allows software teams to build better products, faster. Really what we allow teams to do is separate business logic (who gets what, when, how) from code. There are amazing benefits not just for developers, but for operations, product, marketing and sales, who suddenly can control functionality. As such, we have literally written the book on how to feature flag. [I’ve also written on the wrong way to feature flag] We believe so passionately in the overall movement, that we also open-sourced the book on GitHub. Want to contribute? Open a PR! Excited about helping evangelize feature flags? We’re hiring!

One of the funnest parts of founding a startup is not just building a product, creating a category, and hearing from happy customers – it’s building an amazing team. I am so proud of how much the original eight of us have achieved. But as we continue to grow, it’s clear that we need to have dedicated resources on making sure we’re providing the best customer experience. One of my favorite customer emails read “I can’t believe that the sweet lady who answered the phone and helped me was actually the CEO.”  I’m pleased that Russ Thau, former VP of Sales Intercom, is joining us to build out a team of sales, support and customer success to help our customers get the most out of feature flagging. Interested in joining? We’re hiring!

I hope you see a persistent pattern – we’re looking forward to continuing to build a great product, customer experience and team. A favorite customer email was from Behalf, who said they enjoyed being on the journey with us. Thank you to our customers, team and investors, and I hope you are as excited about our next chapter as John and I are.

19 Dec

A new beginning

Today marks the end of my first week at LaunchDarkly where I’ve been hired on as a DevOps engineer and employee #10.

The current office is great. We’re up on the third floor of the Heavybit building. There’s windows everywhere and so much to see outside. The insides of this office are pretty sweet too. There’s lots of snacks, lots of drinks and the lunches have been pretty good too.

I’ve spent most of the week getting up to speed with the ins and outs of the LaunchDarkly product itself – how it’s developed, deployed and maintained. Also, learning how to use a lot of new tools made by other startups has been pretty interesting.

So far it’s been awesome – I’m surrounded by awesome people who work on an awesome product for an awesome company. I have no doubts that I made the right decision to join and I am excited for all of the opportunities that lay ahead.