Optimizely Web Experimentation vs. Optimizely Feature Experimentation – Which one do I choose?

Optimizely is a well-known name in the digital marketing solutions market. Optimizely’s product suite now includes an even broader range of tools for content management, AI-driven analytics, customer research, and digital experience optimization, reflecting the latest trends in comprehensive customer experience management. But, as the name suggests, Optimizely’s core functionality lies in its ability to optimize, personalize, and experiment. Optimizely offers 2 tools for these tasks – Web and Feature Experimentation.

What are the differences between these solutions and which one should you choose to achieve your business goals?

Optimizely Web

Optimizely Web is an experimentation platform that makes it easy to test with real users, learn about them, and make decisions based on the results.

A/B Testing or Split Testing is a User Experience optimization technique where you split 2 or more variations of the experience between your audience and compare which one is more effective. Testing helps increase conversions, grow the flow of potential customers, and boost engagement.

With Optimizely Web, you can conduct experiments on any channel and device connected to the Internet. The platform includes three solutions: Web Experimentation, Web Personalization and Web Recommendations.

Optimizely Web Experimentation

Optimizely Web Experimentation offers advanced capabilities, including AI-powered A/B and multivariate testing, allowing marketing teams to optimize experiments more efficiently and personalize at scale. The visual drag-and-drop editor enhanced with AI suggestions allows team members with low technical skills to make changes and create variations without developers’ efforts.

How does it work?

To start using Optimizely Web you just need to install a small snippet of Optimizely code in the Head section of your site. The snippet includes your unique Project ID and allows the platform to run on your property.

You can manage users and employees, as well as assign access levels inside the tool.

• Once you have decided on the design of your first experiment, you can create a variation using the visual editor or custom code.

• Determine where testing will take place. Is it only for a specific page? Or does the testing run on the entire site, like experimenting with a navigation menu for instance. Or perhaps you want to test the pages of a particular template – such as applying changes for the product cards of an e-commerce store.

• Set which events you want to track during the experiment. Events are actions that users take, such as clicks, page views, purchases, or form fills. In Optimizely Web, you can use:
  • click events
  • pageview events
  • custom events using custom code for events that cannot be tracked through clicks – such as video and image views, scroll depth, etc.

• Based on the selected events, you can create metrics. For each experiment you run, you have to choose at least one metric. Metrics tell the tool what changes you want to track and see in the reports. When creating a metric, you will also need to indicate which direction of change you would consider success. For some metrics it will be an increase – like a successful completion of purchases. For others, a decrease – for the page exits or abandoned carts. The experiment report and the system’s prediction of the winning variation will be based on the metric you set, so take this step carefully.

• Determine the audience that will take part in the experiment. You can add or exclude specific user segments based on contextual, demographic, and behavioral conditions. For example, you might want a variation to appear only on mobile devices, only for users in a specific region, or only for those who left items in their cart. In Optimizely Web, you can choose from a range of predefined conditions, or create custom attributes for specific queries using custom code.

• Your experiment is ready to run. With the preview mode, you can check all the settings before publishing the test to a real audience.

• On the results page, you can follow the progress of the experiment and get a prediction about the winning variation. Once you reach statistical significance you’ll get results if the experiment was successful and which variation is the winner. The results are calculated based on the metrics you choose, but you can also explore the analytics of other behavioral factors for each variation. You can segment your audience to see if the results differ for certain user groups. Results reports are available for export so that you can present them to your team and management.

• After the experiment is completed, you can implement the winning variation into the live version of the site or continue testing if necessary.

Not all experiments are successful. Sometimes none of the variations show significant changes, or the original version wins out over the newer ones. Don’t take this as a failure. You have received valuable information and you can continue testing with new inputs. As they say, if all the experiments were successful – we wouldn’t need them, we could just go and make changes to the site right away.

Optimizely Web Personalization

Optimizely Web Personalization leverages AI to create even more accurate and scalable targeted experiences based on user demographics, behavior, and predictive insights in real-time. You can use personalization to create a unique and relevant visitor experience and increase conversions and engagement.

For example, you can create special offers for users from different geographic regions based on their national holidays or weather conditions. Or personalize the page titles to match the ad creatives that brought users to the page in the first place.

What is the difference between Personalization and Experimentation?

An experiment is rather a separate project with a certain hypothesis, having a beginning and an end. Personalization is an ongoing process with iterations within.

Starting a personalization campaign is very similar to creating an experiment. The key differences will be in audience setup and results.

You’ll need to prioritize audience segments and experience variations so that the system can understand what experience to present to users that fit multiple segments at the same time.

In the personalization campaign report, you won’t see the variations in comparison. Instead you will get data on how the implemented experience has impacted user behavior.

In order to get this comparison, the system by default shows 5% of visitors the original version. The ratio of users who will see the original and personalized version can be changed at your discretion during the campaign setup stage.

Optimizely Web Recommendations

Optimizely Web Recommendations, powered by AI-driven algorithms, enables more dynamic, behavior-based recommendations on any page and provides deeper insights into user interaction and impact. Recommendations provide an opportunity to engage users to explore more products or content from you, and help to increase engagement, AOV (average order value), up-sells, and cross-sells.

Recommendations can be used both in experiments with Web Experimentation, personalization campaigns in Optimizely Personalization, and as separate standalone Web Recommendations campaigns.

To implement recommendations you will need to create a catalog of items, set up the Recommender mechanism, set filters, and add recommendations block to pages. 

Recommendations are quite obvious for ecommerce websites, but they also work well for B2B Companies with content items such as blog posts, videos, and additional materials.

4 recommendation algorithms

There are 4 recommendation algorithms, but you can only choose one for your campaign:

• Co-browse – this algorithm recommends items similar to what the user is browsing at the moment. For example, if a visitor watches teapots in this session, the algorithm will suggest other teapots.

• Co-buy – Matches complementary items to what the user has viewed. For example, if a person was browsing bikes, the algorithm would suggest bike helmets or baskets.

• Collaborative filtering – this algorithm is based on the combined browsing history of the user and other similar users. The items in the recommendation will be different for each individual visitor.

• Popular – shows the most popular, bought, or viewed items. Use it to showcase the best you have to offer.

The algorithms use different inputs to build recommendations:

• A visitor-based algorithm is based on each visitor’s specific browsing history. The results will be the same no matter what page the visitor is on, but will be customized for each identified user.

• An item-based algorithm is only interested in the page the user is currently visiting. The recommendations will be the same for all visitors, but will vary from page to page.

• A global algorithm doesn’t care about the current visitor or page. Such recommendations are most commonly used on high-level pages (such as the homepage), described as “our most popular choices” or “check out these bestsellers”. This can be a great start for new users of Optimizely Web Recommendations.

Optimizely Feature Experimentation

Optimizely Feature Experimentation allows development teams to run experiments anywhere in your technology stack. With feature tests, you can test whether a new feature is a good fit for your web page or app. 

Special feature flags allow you to turn features on and off without deploying new code and updates to the app store. 

You can work and enable/disable features in multiple environments, such as development and production. You can use the development environment to submit a feature to your QA team or users included in the research; and roll it out to the production environment after all checks have been completed.

With Feature Experimentation, you can also set up gradual rollouts of functionality for all users. For example, start at 25% and increase it if you see improvements, or roll it back if things get worse or you find bugs. 

You can also choose to introduce a new feature only to a specific audience group. For example, offering additional payment options to users in certain regions. Unlike Web Experimentation, FullStack does not have preset attributes to select user segments, you will need to create your own attributes based on custom code.

For each feature flag, you can A/B test variations to explore which feature execution would be preferable for your users. You can test different variations of the user interface, a different algorithm, or a different way of building a feature. As with experiments, you can set up the audiences that will participate in this testing.

Based on the events and metrics tracked, the system will provide you with reports on the impact of certain features on the experience of your users and comparisons of variations. If the feature test completes successfully, you can deploy the feature with the winning configuration to the entire audience.

What is the difference between Web Experimentation and Feature Experimentation?

Both Optimizely Web and Optimizely Feature Experimentation are fundamentally about experimentation. However there are significant differences between these tools, both in terms of structure and in terms of goals and objectives. Let’s analyze the main ones.

Growth Oriented – Product Oriented

Optimizely Web is more of a tool for marketing teams. Primarily, it’s designed to increase the number of conversions and optimize marketing and purchasing funnels with the help of experiments.

The Optimizely Feature Experimentation, on the other hand, is used by product teams who want to experiment with their product and its features without any performance hits. Feature Experimentation users are less concerned about conversions and are working to improve engagement, retention, usage, and lifetime value.

Technical Skills

Optimizely Web Experimentation does not require high technical skills from its users. Without a single line of code and with the help of a visual editor, you can implement your hypothesis and conduct testing.

Optimizely Feature Experimentation is a more complex tool that requires experienced developers to deploy new code to conduct server-side or client-side experiments.

Snippet vs. SDK

In order to implement Web Experimentation and get started, you just need to put a small snippet of code in the page where you want to start experiments.

Feature Experimentation works through SDKs (Software Development Kits) that are available for all popular development languages.

Fast or Deep

If you want to quickly optimize your website’s interface, test layouts, colors and texts – everything related to appearance – you should turn to Web Experimentation. This tool does not require coding and releases and allows you to conduct many experiments at high speed while measuring your results.

The Optimizely Feature Experimentation approach allows you to test more complex algorithms, basic functions, and business transactions. You will be able to learn how your product works, but it will take time to roll out the experiment. You will need to adhere to standard code release practices such as code review and testing.

Single channel vs multi-channel

Optimizely Web Experimentation is typically used to run experiments in a single channel, most commonly your company’s website. Optimizely Feature Experimentation allows you to experiment with multiple channels simultaneously, such as the web, mobile devices, SMS, and email.

Performance

Although the Feature Experimentation experiments are more complex, you may notice slightly better performance with them. This is because Feature Experimentation supports server-side experimentation, which improves performance and avoids page load impact and flicker.

Which one do you choose?

Which tool to choose depends on your goals, objectives, and the product you plan to test.

If your team lacks the technical expertise to set up Optimizely Feature Experimentation or enhance Optimizely Web, you can turn to Optimizely Certified Partners who provide developer services to Optimizely clients. 

First Line Software has been an Optimizely partner for 14 consecutive years.

Our engineering team has completed over 350 Web Content Management (WCM) projects, utilizing more than 40 certified engineers and 200+ architects, developers, quality assurance technicians, UI/UX designers, and project managers. 

Learn more about First Line Software’s Optimizely Implementation Expertise or tell us about your project and get a free first consultation from our experts.