Are you looking for ways to improve your user experience and increase conversion rates? While A/B testing is a great way to test different surface variations, feature testing may be a more appropriate approach to experimenting with more complex ideas that require nuanced testing. In this blog post, we’ll explore the importance of feature testing and uncover what it is, why it’s beneficial, and how you can get started with feature testing in Optimizely. We’ll provide you with some practical tips and best practices for conducting effective feature tests that can help you improve your performance and achieve your business goals. Whether you’re new to feature testing or looking to improve your existing strategy, this post is for you. So let’s dive in!
What Are The Features?
In software development and website development, a feature refers to a specific functionality or capability of a product. It can be a component or an aspect of the software or website that serves a specific purpose or provides a specific benefit to the user. Features can be simple, such as a button that performs a specific action, or complex, such as a multi-step checkout process that involves several different interactions with the user. The development of features is an essential part of creating a product that meets the needs and requirements of the users and can play a critical role in the success of the software, or website, and ultimately your business.
What Are Feature Tests?
Feature testing is the process of testing individual features or components of a software or website to determine their impact on user behavior, engagement, and conversion rates. By testing different variations of specific features, developers and product teams can gain insights into which variations perform better, identify user pain points, and optimize the product for improved performance.
How Feature Tests Work?
Feature testing works similarly to A/B/n testing, allowing you to test variations of a feature by toggling its flags on and off. You can experiment with having or not having the feature available, test different ways of how it’s designed and built, and align different metrics to measure within your feature’s variations.
As visitors interact with your website or application, the feature test collects engagement metrics to determine the impact of it on user behavior. You can then analyze the statistics on an analytics dashboard to evaluate the effectiveness of the feature and determine which configuration of the feature is the most successful.
Benefits of Feature Tests
Determining if a feature fits your product
You can use feature tests to validate whether a new feature is a good fit for your product. At the same time, you can also determine what variation of it will be most successful. If the
feature test finishes with a positive result, you can then roll it out with the configuration of the winning variation of the feature test.
Determine the best feature configuration
In addition to enabling/disabling your feature at the variation level, one can also run a test to experiment on different configurations and test scenarios for the feature. That way it will be possible to determine the best feature experience.
Discover bugs early in a real environment with low-risk
In addition to standard software testing practices such as unit testing and functional testing, feature testing can be a good way to discover bugs and issues that may only arise in edge cases of your production environment with real users. By wrapping new features in feature flags, you can quickly roll back any bugs without extraneous code deployment.
Feature testing helps you to test and optimize the quality of your product by delivering the feature to a small group of users in the very first stage to see how it works, and to detect issues. That way if any problems are found, they only impact a small group allowing you to still roll out quickly without waiting for new code to deploy.
Feature Tests and Optimizely
There are different platforms that help developers to apply feature tests in easy ways. One of the most robust and widely known tools is provided by Optimizely called, Optimizely FullStack.
Optimizely Full Stack is a feature flagging and experimentation platform for websites, mobile apps, chatbots, APIs, smart devices, and anything else with a network connection. You can deploy code behind feature flags, experiment with A/B tests, and roll out or rollback features immediately. All of this functionality is available with zero performance impact via easy-to-use SDKs.
Optimizely offers a management user interface that enables you to configure feature flags, set up A/B test rules, and define rollout rules for feature flags. The interface is user-friendly and allows you to easily view experiment reports to make informed decisions after running A/B tests on specific feature flags. Additionally, you can define target audiences and track events to collect data for measurement purposes.
Optimizely provides a strong SDK to allow you to implement new features based on a feature-testing strategy. This SDK is for developers and is written by various programming languages like Android, C#, Java, Go, JavaScript (Browser), JavaScript (Node), PHP,
Python, React, React Native, Ruby, and Swift. The robust SDK allows you to easily apply feature tests to any kind of application such as mobile, desktop, web, and others.
Is there any performance impact when using Optimizely FullStack in implementing a new feature?
Optimizely claims that its platform has zero or microsecond performance impact, which would make it a perfect solution for feature testing in terms of performance. However, it’s important to analyze how performance is impacted when using the platform.
There are several steps that developers need to follow when applying feature tests to feature development using Optimize Full Stack, including:
- Choosing the suitable Optimizely SDK for their application’s programming language
- Initializing the Optimizely Client instance
- Getting decisions based on user context and feature flag key
- Adding business code based on the retrieved decision
- Sending tracking events.
In addition, there are various methods of deployment and some of them can have the potential to impact performance because they involve direct interaction with the Optimizely Portal to get and send information. Currently, Optimizely SDK supports sending tracking event information by pushing them into the queue before actually sending events to the Optimizely Portal via the scheduler. This has the potential to improve performance significantly, but you’ll need to give it some time before you can see the latest experiment report. If you want to see real-time reporting you can use Event API directly to send events immediately. That way, you will be able to see the latest report quickly but it will affect overall performance.
Your developers will need to find an optimal balance between latency and accuracy, performance, and real-time reporting.
Common scenarios to use Optimizely FullStack
- You need to implement a new feature and you want to be confident that this feature works well and bug-free when released to your audience.
- You want to implement a new feature but you do not want to release it to all of your users and want it to be accessible to only a targeted group.
- You want to implement a new feature with some variations, while also releasing different variations of the feature to different target users.
- You need to implement a new feature, but there are a lot of options on how to build it and you are considering which option is the best fit with your user’s needs.
- You want to change a current feature and you are not sure that this new version is a better version than the existing one. Here you can use feature tests to compare and measure results between old and new versions to make an informed decision.
Some Other Feature Test Platforms
There are other feature testing platforms on the market that allow developers to manage and test new features before rolling them out to all users. While they share similar goals, there are some differences between them.
Launchdarkly
Launchdarkly is a cloud-based feature management platform that enables continuous delivery through feature flags. With feature flags, developers can release new features to a subset of users and monitor the results, allowing them to quickly iterate and improve their products. Launchdarkly offers a range of integrations with other tools, including popular software development platforms like GitHub and Jira.
Split
Split is another feature management platform that helps developers test and release features in a controlled manner. One key difference between Split and Launchdarkly is that Split offers a range of testing methodologies, including A/B testing, multivariate testing, and feature rollouts. This allows developers to choose the approach that best suits their needs and get more precise results.
Flagship
Flagship is a feature flagging and A/B testing platform that offers a range of tools for testing and optimizing features. One key feature of Flagship is its ability to segment users based on a wide range of criteria, including demographics, behavior, and location. This allows developers to test features with specific user groups and get more targeted insights.
Statsig
Statsig is a feature-testing platform that provides developers with real-time data on how their features are performing. It includes a range of tools for testing and monitoring features, including A/B testing, feature flags, and user segmentation. One key feature of Statsig is its ability to provide real-time insights into how users are interacting with features, allowing developers to quickly identify and fix issues.
In summary, while all four of these platforms share similar goals, there are some differences between them. Launchdarkly focuses on continuous delivery through feature flags, while Split offers a range of testing methodologies. Flagship allows for targeted user segmentation, and Statsig provides real-time insights into feature performance. Ultimately, the choice of which tool to use will depend on the specific needs of the development team and the goals of your project.
What’s Next?
Feature testing allows developers to improve the quality of their products, either by modifying already existing features or when introducing new features that would be more readily accepted by users.
With feature testing, you can take small risks while still minimizing the impact of features on your user base by enabling it for a certain portion of your traffic and disabling it for the rest. If the feature yields positive results, you can roll it out to everyone else to maximize its value.
Feature testing using feature flags is a powerful tool that can make all the difference in your business’s success, but only if you apply it correctly.
You must clearly understand how to build experiments in relation to your business goals. There is no point in just changing button colors and wasting time and money on meaningless tests. Your strategy for experimentation should be clearly calibrated. New features that you introduce into testing should be high quality, without errors, and always take into account the goals of your users. What’s the point of testing a feature if you already know it’s not likely to work?
If you’re new to experimentation, it’s a good idea to have an expert team to support you. First Line Software has extensive experience in the CMS, DXP and Commerce arenas and is proud to be an official Optimizely Implementation partner. Our experts are always ready to discuss your business needs to find out how today’s technologies and tools can help you. Contact us for a complimentary consultation.
Binh Nguyen
Optimizely Developer at First Line Software
Binh is an expert in EpiServer/Optimizely framework passionate in programming in a wide range of frameworks, enjoing challenges, willing to learn new things and improve herself day by day. Working in a lot of ecommerce projects from small to big scope, Binh has experience in improving performance for big ecommerce system based on .NET stacks and using EpiServer/Optimizely to build them up. Binh runs her blog on Optimizely and participates in the OMVP (Optimizely Most Valuable Professionals) Program.
Clinovera
