Month: March 2019

An experimenter’s guide to marketing technology

Most business leaders today are familiar with Scott Brinker’s famous Marketing Technology Landscape supergraphic:

In 2018, the graphic included 6,829 marketing technology solutions—almost double the number of solutions depicted in 2016. There are literally thousands of technology platforms at your fingertips. And each promises to solve every one of your business pain points.

But technology without purpose and strategy quickly becomes shelfware. A strategic approach to both sourcing and leveraging your marketing tools is essential.

Today’s post details the technology roadmap recommended by Widerfunnel’s Senior Experimentation Strategists. This roadmap highlights the “why” behind implementing certain technologies at certain moments in your organization’s experimentation journey. As well as how to get the most out of these tools by taking a strategic approach.

Note: We will not cover every type of marketing technology, but will focus on the tech stack that should be leveraged by a team that is conducting online experiments.

Strengthening the Technology pillar in your Experimentation Operating System

But first, let’s zoom out.

Your experimentation technology stack is just one piece of your Experimentation Operating System™ (EOS). There are four other pillars, which include: Process, Accountability, Culture, and Expertise.

You must view your tools within the context of the entire system:

• How do they support you in documenting, unifying, disseminating, and automating your experimentation processes?
• How do they support you to hold your experimentation program accountable—to properly track and report on program success?
• How do they support you in collaborating with the organization at-large? In promoting visibility and transparency around experimentation to create a culture of experimentation?
• How do they support your team of experts in terms of usability, productivity, training, and customer support?
• How do they integrate and work with your existing technology stack?

As your experimentation program matures, you will likely experience constraints across the different pillars. Technology will not always be your most important constraint. In fact, we speak to many organizations with extremely sophisticated marketing technology stacks that are floundering in areas like process and culture.

Your experimentation program will only reach its growth-driving potential if it is strong across all pillars.

But today—we are talking technology. If this is your organization’s primary focus or suspected weak spot, you are reading the right blog post.

Technology in the Initiating Phase: Analytics & an experimentation platform

Organizations in the Initiating Phase of experimentation maturity are just getting started. In this stage, an Experimentation Champion is likely working to implement the appropriate technologies and get initial wins to prove the value of an experimentation program.

If your organization is just getting started with experimentation, your primary focuses should be:

1. Ensuring you have a solid analytics foundation, and
2. Identifying and selecting the experimentation platform for your needs

Analytics and quantitative research

Clean data for quantitative research is an essential starting point for any experimentation program.

Analytics data will inform the initial story of your customer journey; it will enable you to prioritize test areas and gain insight into your customers’ goals and intentions. We group these into two buckets: Primary analytics and supplementary analytics providers.

Primary Analytics: There are really two key players that provide primary analytics reporting for most organizations: Google Analytics and Adobe Analytics. Together, these two have a lock on around 75% of the market. Google dominates for websites of all sizes, however Adobe can’t be discounted, particularly as a solution amongst high-traffic Enterprises.

Another company to keep on your radar is Heap Analytics. Heap has emerged as a powerful solution known for its ease of implementation and ability to auto-tag all site activity. For companies with limited IT resources, this single-line of javascript implementation is an attractive option.

Supplementary Analytics: While Google and Adobe dominate as primary analytics solutions, there are many emerging technologies you can use to supplement the data your primary tool is collecting. These tools provide advanced solutions for data visualization, marketer-friendly tagging, and prescriptive analytics.

These solutions may not be worth the expense for smaller organizations in early stages of experimentation maturity. But when you start asking very complex questions and dealing with larger data-sets, your primary data analytics tool can become restrictive.

Companies that are looking to take their analytics game to the next level should look into:

With a solid analytics set-up, your team can start to uncover the low-hanging fruit within your digital experience and identify opportunities for testing. You will quickly find, however, that you need to implement an experimentation tool to increase the confidence you have in the changes you are making.

Experimentation Tool

The next piece of the puzzle is a platform to facilitate experimentation. These tools allow you to measure the statistically valid effects of any change you make by limiting the impact of time and audience variables.

When you rely on analytics alone, you are relying on a less than accurate “before and after” measurement system. Making key business decisions based on this type of system can often lead you down the wrong path.

An experimentation tool allows you to run experiences simultaneously to randomized samples of your audience, giving you data you can trust to make decisions.

Before choosing a tool, it is important to identify your program’s requirements and parameters. For instance:

• What sort of targeting requirements does your website have?
• Are you going to require server-side testing?
• Does your development environment allow for third party Javascript?
• Are you testing on the web or do you need to be able to test in a native application?
• Do you need Single Page App (SPA) support?

The answers to these questions in combination with your budget should help you select the right tool for your needs.

In the North American market, there are four major players in the experimentation space and many niche options. Each has unique strengths and features to offer depending on the needs of your company.

Optimizely

Optimizely is the market leader by market-share and is considered a thought leader in the experimentation space. They lead the way with their Stats Engine, broad integrations, and expansive product suite; they are considered a one-stop solution for organizations in any phase of experimentation maturity.

Optimizely’s experimentation-first perspective means they are usually first to market with the most innovative feature developments and advancements in the industry, keeping their customer base ahead of the competition. If you are an Enterprise organization, this is the first tool to evaluate. These features do come with a higher price tag though, and Optimizely may be out of range for smaller organizations that are just getting started.

Adobe Target

Also servicing enterprise companies, Adobe Target provides a testing solution to those that are heavily leveraging the Adobe technology stack. Like Optimizely, Target offers solutions to most of the key requirements any company may have. Those with Adobe Analytics and other Adobe products may find additional value due to Adobe’s cross-product integration. However, Target isn’t the most user friendly tool. And if you aren’t a trained data analyst or statistician, its statistical model can leave room for error. If you don’t already leverage the Adobe suite, you will likely not get the full value of Adobe Target.

VWO

VWO is a great solution for smaller or mid-market companies. This tool has a much more friendly price-point and allows for all basic client-side testing functionality. They have a strong SmartStats Engine, and built-in variation heatmaps, which are a major bonus. VWO is a great option for organizations with a standard client-side website environment that are looking to prove the value of experimentation. However, companies with very specific technical needs may want to pass on VWO due to their lack of server-side or SPA technology.

Google Optimize

New to the game (sort-of), technology giant Google has also developed an experimentation tool. The obvious advantage of Google Optimize is its native Google Analytics (GA) integration, and its free entry-level price-point. That said, the free tool is rudimentary and lacking in most advanced features. It also limits users in the number of experiments and metrics that can be in the tool at once. Optimize can be a worthwhile solution for GA organizations that are just getting started with basic testing.

Google’s paid version, Optimize 360, unlocks additional experiments and audience criteria that allow for more advanced experimentation. But it is only as valuable as your GA integration. If you have a team of GA experts, then you can do some creative analysis. However, Optimize 360 is likely not worth it for more entry level programs.

Keep in mind, Google Optimize is a sleeping giant. If Google decides to pour resources into the development of this tool, it could quickly become the dominant player in the space.

Sentient AI

If your website is well-suited for multivariate testing (MVT)*, then Sentient AI offers a very interesting solution. Their tool uses Learning Evolutionary Artificial Intelligence (LEAF).

Loosely following the Theory of Evolution, Sentient lets you set thousands of variables and then watch as the tool “breeds” new combinations of variables and sends others to “extinction”. Eventually, it works its way towards the optimal combination. Due to its complexity, this solution is not for everyone. But in the right hands and on the right site you can cover a lot of ground quickly.

*Sites suited for MVT often have many fixed, modular components that are independent of one another.

Client-side versus server-side experimentation

One of the most important questions to consider at this stage is whether or not you need a client-side or server-side solution. Both have benefits.

Client-side execution means the code for your experiment variation will be injected through the browser. While this can be worse for performance, it enables the use of simple WYSIWYG interfaces that allow marketers to make changes without involving developers. One of the other major drawbacks is that you can only test on features that already exist in the DOM.

Server-side execution means that the variation code will actually run on your server. The primary benefits of server-side are improved performance, security, and the ability to test features that do not exist in the control environment. Teams can leverage this to roll out new pages, entirely new functionality and prototypes, or make changes on pages with complex server calls.

Server-side also greatly reduces the time needed to hardcode experiments because the code is already built in your native environment and follows all of your conventions. Server-side experimentation does require more initial investment to install the appropriate SDK but has efficiency and security benefits down the line.

If you are evaluating server and client-side technologies, consider the expertise of your engineers. Because server-side testing is done in your native environment, your team may find it more accessible. Client-side, on the other hand, relies heavily on Javascript and jQuery.

Technology in the Building Phase: A foundation for collaboration

Organizations in the Building Phase of experimentation maturity are bought in on the value of experimentation. In this stage, an Experimentation Champion or team is likely establishing process and building the infrastructure to scale the program.

An analytics tool and an experimentation tool make up the technology stack of a basic Experimentation Operating System. A single team can leverage both to plan, run, and analyze experiments. However, as you work to scale your experimentation program, collaboration becomes crucial. How can you enable experimentation across many teams and business units?

While an individual can test with little documentation, your organization will need solid project management and record keeping to scale up an effective program. In our experience, you will need a specific experiment collaboration system in place to move upward on the experimentation maturity scale.

There are a few things to consider when selecting collaboration tools:

• How will people throughout the organization submit test ideas?
• Where will experiment wireframe and design files be hosted?
• Where will people communicate about an experiment?
• How will the current status of each experiment be displayed?
• Where and how will the results be stored?

You may not need to source technologies that address all of these questions at the outset, but you should consider tool(s) that will be able to grow alongside your program. Here are a few popular solutions we have seen:

JIRA: JIRA can really do it all. Although its interface can be confusing at times, JIRA is extremely flexible, allowing you to build a custom process that works for most situations. JIRA is a strong solution if your experimentation program sits within Engineering or Product, since it is likely JIRA is already the tool of choice of your engineers. If your program sits within less technical teams, such as Marketing, you will likely want to turn to something a little bit more user friendly.

Asana: Asana is a strong task management solution, and is the tool of choice for project management at WiderFunnel. It allows users to build consistent templates, facilitates task assignment and communication, and has useful scheduling features. The accessible interface makes this a tool that everyone in the organization can use with ease.

Trello: Kanban boards! Certain organizations love working in a kanban view, and Trello is the leader in this space. For companies that want a visual representation of their experimentation projects, Trello can be a great solution. Plus, it integrates well with JIRA as both come from the same parent company—Atlassian.

Optimizely Program Management: One of the key value propositions offered by Optimizely is the integration with their Program Management solution. Optimizely Program Management allows users to store experiment ideas, vote on priority, track insights, report on program success, and more. Although pricey, Optimizely Program Management is a glimpse into the future of experimentation and insight management.

Traditional Spreadsheets: For teams in the early days of the Building Phase looking to get organized, spreadsheets are definitely a viable option. While spreadsheets often fall short on image storage and communication, they are a free, easy solution for smaller teams tracking a simple program. Keep in mind that scale will likely be a problem here.

As you add more tools to your marketing technology stack to enable experimentation, you’ll want to make sure they integrate nicely. If you’re an Adobe user, your analytics and testing tool will already be integrated—you should ensure that anything else you layer on plays nicely with this suite.

As an alternative to suite solutions, several leaders in the experimentation tech space have formed the Digital Experience Stack (DXS). If you use Optimizely or another tool within this stack, you will want to evaluate other DXS solutions.

Achieving experimentation maturity with qualitative research

Organizations in the Collaborating Phase of experimentation maturity are expanding the experimentation program and collaborating across teams. Finalizing a communications plan and overall protocol for the program is a priority here.

For organizations in the Scaling Phase of maturity, experimentation is a core strategy. Standards are in place and success metrics are aligned with overall business goals, enabling testing at scale.

Game-changing experiment ideas come from customer research. Quantitative input from analytics will help you identify potential pain points within your digital experience, but it only tells a portion of the story. Sophisticated experimentation programs also work to layer in qualitative research—an equally important counterpart that will help you fill in gaps in your understanding of your customers.

The goal with qualitative research is to uncover the “why” behind the “what” that you have observed with your quantitative tool.

Ideally, your experimentation team will have all of the following qualitative tools at their disposal:

• Scrollmaps
• Clickmaps
• Movemaps
• User polls
• Surveys
• User session recording
• Interviews

User Engagement Tools

Many organizations start with user engagement tools. These tools—including scrollmap, clickmap, heatmap, and user session recording features—help you visualize the visitor experience. They are useful because they are passive (requiring no additional action from your visitor) and are often low cost and easy to use.

When analyzing data from these tools, you’ll want to consider:

• What % of your visitors is seeing specific important content?
• What % of your visitors is scrolling past the fold?
• Is there a particularly steep drop off after seeing a specific piece of content?
• When comparing multiple calls-to-action, which are more commonly clicked?
• What valuable information might your visitors be missing?
• Where should you run your next test? (You may not want to test an element that few users are seeing)

Polls & Surveys

Polls and surveys take user research a step further by actually addressing specific questions to your audience. They give you the opportunity to ask questions while your user is in the shopping mindset and is evaluating your product. These require some action from users, but can often uncover much deeper insights (such as pain points within the customer journey you may have overlooked).

Exercise discretion with polls and surveys. Although they can provide rich customer feedback, they can also distract from your primary conversion goal.

Recommended tools to evaluate include:

Advanced experimentation & personalization: Data and customer management technologies

We hear shouts of “hyper-personalization” constantly—a one-to-one customer experience is the pinnacle for many organizations today. Of course, this relies on the existence of underlying data to define what makes an experience “personal”. Most organizations do not have the proper technology in place to enable this.

If your business has been struggling to do effective personalization, you should 1) interrogate your overall personalization strategy, and 2) look closely at how you are managing your customer data.

The proper use of a robust customer data platform (CDP) is a key component that differentiates mature experimentation programs from the immature. These platforms open up the world of audience management, boosting your ability to identify and target high value audience segments and plan test strategies around these groups.

Customer Data Platforms (CDP)

The CDP Institute defines a Customer Data Platform as “packaged software that creates a persistent, unified customer database that is accessible to other systems.”

A major benefit of a CDP is the ability to deliver a more effective customer experience and more impactful marketing messaging. (Which is really the goal with personalization). Your customers want a consistent experience across all of the channels and devices they’re using. They don’t want to see an ad for something they have already purchased. A CDP allows you to gain a complete view of your customer and deploy a consistent experience across touchpoints.

It is important to note, however, that a CDP is only as useful as it is comprehensive and actionable. This means that the number of data sources available to your CDP, as well as the number of execution integrations are both critical.

If you aren’t collecting data from multiple sources—website, mobile app, customer service system, in-store behavior, beacons, etc.—you may not be ready for a CDP. As well, if you can’t activate this data across multiple touchpoints—website, in-store, customer service interactions, etc.—you will not unlock the full potential of a CDP to provide a truly unified customer experience.

A CDP is not a personalization tool. However, it provides the data that will allow you to get the most use out of your personalization and experimentation tool.

If yours is a mature program and you are looking to enhance your personalization efforts or improve your overall data management efforts, Tealium’s Universal Data Hub is a great choice. Evergage is also a strong solution, combining a CDP with real-time personalization capabilities. If you already have a data platform in place, Evergage also functions as a powerful standalone personalization platform.

If you are looking for other tactical personalization solutions, you should also evaluate Optimizely Personalization and Dynamic Yield.

—

The business world has come a long way from the Mad Men era of focus groups and hunches. Today, technology enables experimentation at scale. It allows organizations to test one experience against another, achieving a statistically confident result. Which greatly reduces risk in decision-making.

However, it is vital that you define your experimentation strategy and ultimate objectives before sourcing your technology stack. Choose tools that fit these objectives, as well as your organization’s culture and the skill levels on your teams.

A final note: Before sourcing more tools, make sure you have a clear idea of what your program constraints really are. You can install every tool on the market, but that doesn’t mean you will have an effective experimentation program. First, you need a map of how you plan to get your program from A to B; technology is simply what you will use to pave the roads along the way.

What tools make up your marketing stack for experimentation? How do these tools support your overall experimentation efforts? Are your favorite tools on this list or did we miss them? Leave your thoughts in the comments section below!

Author

Michael St Laurent

Director of Experimentation Strategy & Product Development Lead

Contributors

Natasha Wahid

Marketing Manager

Enjoy this post? Share with your friends and colleagues

Share on Facebook