Changing organizational culture and fostering an experimentation mindset

Changing organizational culture and fostering an experimentation mindset

Experimentation fuels growth. Organizational culture supports experimentation.

Experimentation is a core strategy for product development and growth at organizations that are winning in the market.

In a report on the Insights-Driven Business, Forrester demonstrated that businesses with closed-loop learning processes at their core are growing at least eight times faster than the global GDP. These learning processes have experimentation at their center.

Closed loop learning process Forrester
Source: The Insights-Driven Business, Forrester, 07/27/2016

Leading companies like Netflix, Uber, Amazon, Airbnb, Microsoft are quick to share that they are fuelled by experimentation. These are what we would call mature organizations—they leverage experimentation to generate continuous insights and growth that impact their bottom-line.

Pursuing experimentation maturity

At WiderFunnel, we have been working with brands to build and scale insight-generating experimentation programs for over 12 years. In doing that work, we have identified five phases of experimentation maturity that organizations progress through.

Experimentation maturity levels Widerfunnel
WiderFunnel’s five phases of experimentation maturity.

Level 1: Initiating Organizations at this stage are just getting started. An Experimentation Champion is working to get initial wins to prove the value of an experimentation program.

Level 2: Building In this stage, an organization is bought-in on the value of experimentation and an Experimentation Champion or team is establishing process and building the infrastructure to scale the program.

Level 3: Collaborating Organizations at this stage are expanding the experimentation program and collaborating across teams. Finalizing a communications plan and overall protocol for the program is a priority here.

Level 4: Scaling Experimentation is a core strategy for these organizations. Standards are in place and success metrics are aligned with overall business goals, enabling testing at scale.

Level 5: Driving The highest level of maturity. Experimentation is the organization’s growth and product strategy. The Amazon’s, Netflix’s, and’s are here.

The organizational culture component

While there are multiple pillars of a mature experimentation organization—such as a powerful technology foundation and clear objectives for the program—developing a culture of experimentation is essential. In order to move from one phase into the next, you must foster an organizational culture that embraces testing and learning.


Experimentation at scale: A roadmap for the enterprise

A culture of experimentation is one very important component of a successful experimentation program. However, there are other factors to consider, including organizational structure, program KPIs, and hiring choices. This guide provides a step-by-step roadmap to get you where you need to go.

Ultimately, a culture of experimentation is a factor of effective communication. If you can’t get this piece right, your organization will never reach the highest level of maturity.

A hypothetical example

Imagine you are an Experimentation Champion trying to build a testing program from scratch. You have a vision for experimentation; you believe in its value and are excited to get moving. You are a one-person show, but you are convinced you can get the rest of your company on board.

To do this, you decide to get everyone involved in the experimentation program at the outset. You open the program up to the whole organization. You start sourcing ideas from everyone and everywhere.

But there’s a problem: Not everyone understands experimentation. People are contributing ideas without thinking about them, without thinking about the greater context. Soon, you are buried under ideas you can’t actually execute on.

And the experimentation program becomes a joke; a dumping ground where ideas go to die and nothing seems to be getting done. The teams around you lose faith and gradually stop submitting ideas. Eventually, experimentation becomes…

Something we tried once. It doesn’t work for us.

We see this story unfold all too often. But this doesn’t have to be your story. You, as an intrapreneur, can foster a culture of experimentation and avoid this crash and burn scenario.

Inspire. Educate. Inform.

You can do this by leveraging three essential actions in your communication: Inspire, Educate, and Inform.

Inspire means creating the spark. Inspiration occurs when you create a moment of clarity and awareness of new possibilities, as well as a desire to take action—to get involved.

Inspiration […] involves a moment of clarity and awareness of new possibilities. This moment of clarity is often vivid, and can take the form of a grand vision, or a “seeing” of something one has not seen before (but that was probably always there). Finally, inspiration involves approach motivation, in which the individual strives to transmit, express, or actualize a new idea or vision. [It] involves both being inspired by something and acting on that inspiration.

Educate means training. This action involves training, by instruction and/or supervised practice, in a particular skill. In this case, how to do experimentation.

Inform is closing the loop. Inform means actually communicating a message or making something known.

change organizational culture actions
Note: While these three actions are pictured in a linear fashion, they are often occurring simultaneously.

In the journey to a mature culture of experimentation, these three actions will need to be at play at different levels.

When your organization is just getting started with experimentation, testing is likely owned by a single, core team. Inspire, Inform, and Educate must be at work in this core team before your organization can move into the next phase.

culture change core team

To scale, your organization will need to empower supporting teams to participate in the experimentation program. The ultimate goal being an organization where every single person has an experimentation mindset, from your CEO to Lead Engineer, to the customer support heroes who pick up the phone everyday.

scaling experimentation organizational culture

To get here, you will have to continuously Inspire, Educate, and Inform, to drive organizational change and foster the experimentation mindset. But what does that actually look like?

Just getting started with experimentation

In the early stages of maturity, a core team is often responsible for running optimization experiments. They are likely focused on getting initial buy-in for testing and building momentum around positive results.

One or a few team members should also be focused on Inspiring, Educating, and Informing necessary stakeholders, to lay the culture of experimentation foundation.

Inspiring your core team

As an Experimentation Champion, your first priority should be to recruit a core experimentation team. This team should include an Executive Sponsor (if that isn’t you), and individuals or partners who can execute experiments: Design, Engineering, Data Science, Experimentation Strategy, etc.

You may have these resources in-house, or you may decide to bring in an enabling partner to augment your capabilities. Either way, your core team should help you 1) develop and prioritize experiments, 2) execute experiments, and 3) socialize experiment results.

You will need to inspire the members of your core team to motivate them to get involved and stay involved in experimentation. At the outset, this means tailoring your message to each individual and showing them the new possibilities of testing. You should constantly ask yourself:

  • Who am I speaking to?
  • What do they care about?

A real-world example

One of our clients is a technology startup. Several months ago, the Head of Demand Generation decided to implement an experimentation program. She knew she needed to start by recruiting a small core experimentation team.

She began with an Executive Sponsor—the company’s VP of Revenue. This VP has an allstar sales background, but wasn’t familiar with the concepts of “experimentation” and “conversion optimization”. She needed to show him new possibilities that would matter to him.

The company had just finished a website redesign. In this context, the Champion worked carefully to explain to her VP that, while the project had succeeded from a brand and aesthetic angle, there were still potential points of friction in the user experience.

She pointed out that the company was spending substantial money to funnel traffic to the redesigned website, and emphasized the missed opportunity of not addressing these potential barriers to conversion—the opportunity to increase their primary metric by 2%, 5%, 10%.

She spoke to the VP in financial terms that mattered to him, and showed him the financial possibilities around marketing experimentation. And he got on board because he was inspired.

Educating within a core team

At this stage of experimentation maturity, Educate and Inform can often be done informally.

Your core team should have the skills to do experimentation, but they may not have complete understanding around why and how to do it. Design may understand UX best practices, but they may be wary of marketing experiments that could challenge brand standards. Engineering may be highly focused on product development, and lack the front-end development experience needed to develop marketing experiment variations.

Your best course of action in this case is to involve your Designer(s) and Engineer(s) in the conversation as early as possible. Remember, these are members of your core team. As such, they should be a part the experimentation conversation from start to finish.

This is education via supervised practice: by doing. As the Experimentation Champion, you should be guiding the conversation around overall objectives and experimentation frameworks. You should be educating the other members of your core team.

If you yourself are unsure about the in’s and out’s of testing, it may be a good idea to bring in an experimentation partner. With our technology client, the Champion knew she was missing critical pieces of a core team, including strategic support and dedicated resources. Which is why she brought in WiderFunnel as an enabling partner. In that role, we are able to work with her to transfer knowledge around experimentation to the members of her core team.

Informing within a core team

When it comes to Inform, you must make sure that you are closing the loop with each stakeholder on your core experimentation team.

This means informing your team members when an experiment is launched and informing them when and why it is completed. It means including them in the results analysis conversation and in determining next steps.

If someone is involved in an experiment, you must keep them informed, particularly regarding the impact of that experiment. Whether this is via email, Slack, or simply a face-to-face conversation, the importance of closing the loop cannot be overstated. Because nothing is quite as motivating as seeing the bottom-line impact of your work. And nothing is quite as de-motivating as contributing to a project and not knowing the results of your contribution.

One of the most important things to note at this stage is not to overreach. Focus on recruiting the core team and resources needed to get your experimentation program rolling. Work to Inspire, Educate, and Inform these key people—Engineering and Design, your Executive sponsor, and your Executive team.

As you scale the experimentation program and begin to empower supporting teams, your core team will need to Inspire, Educate, and Inform these supporting teams to get them up and running.

Of course, one person can’t shoulder Inspiring, Educating, and Informing for the entire organization. To support scale, you will want to implement systems that help to automate the actions of Inspire, Educate and Inform.

Building momentum for experimentation and driving organizational change

So what does that look like? Let’s look at a slightly more mature organization.

Another partner of ours is a large, digitally mature financial services company. When we partnered with this company, there was already Executive-level buy-in for testing, as well as a general understanding of the value of experimentation.

The core experimentation team had been assembled. It consisted of an Experimentation Champion, an Executive Sponsor, and WiderFunnel as an enabling partner. The function of this core team was to enable supporting teams (rather than to execute experiments). As an organization matures, this is often the role that a core experimentation team moves into—a facilitating, enabling role rather than an executing role.

At this particular organization, the core experimentation team was trying to enable eight different product marketing teams to develop and launch digital experiments.

Taking advantage of opportunities to Inspire your organization

In this case, the Experimentation Champion had taken advantage of an opportunity to inspire the larger organization.

One of the highest visibility product teams at this company had just gone through a page redesign, which was performing terribly. To address this, the Champion brought in WiderFunnel to analyze the redesigned page, identify potential barriers to conversion, design a variation, and launch an experiment to try to improve page performance.

The core team was very confident that the experiment would win because the redesign was performing so terribly. There was a lot of potential. And they were right—the variation performed much better than the redesign. Because this was such a high-visibility product, the whole organization was watching.

The lesson here? Be opportunistic about promoting the experimentation mindset. The other seven product teams saw, first-hand, the new possibilities associated with experimentation. And they were chomping at the bit to get started; to get involved.

But first things first.

At this stage, documentation becomes critical. When one small team owns and operates experimentation, you can get away with little to no documentation. But as you expand into supporting teams, centralized documents, processes, and standards become necessary to enable proper knowledge transfer. Documentation is one of the systems that helps to ‘automate’ the actions of Inspire, Educate, and Inform.

Enabling Inspiration, Education, and Informing at scale

To do this, the core team at this organization decided to leverage two primary activities: workshops and documentation.

Documenting experimentation standards.

Workshops became the primary vehicle for education and continued inspiration; documentation became the primary guide for how and whom to inform.


While the Champion in our previous example was educating her core team informally, via practical instruction, this organization needed to take a more formal approach to educate eight separate teams.

The core team needed to transfer knowledge around how to run experiments, but perhaps more importantly, around how to think about experimentation. The supporting teams had a very narrow view of ‘experimentation’. It was seen as a UX tactic—testing small tweaks to improve a particular conversion metric.

We wanted to educate these teams on the true potential of experimentation—how to use it to get real answers to real questions. To do this, we designed a series of workshops that walked these teams through various questions that they could and should be asking when developing experiment hypotheses.

There were questions that asked teams to refocus on their overall objectives and contextualize any experiment ideas within their broader goals.

There were questions that asked teams to refocus on their website visitors and customers and develop ideas based on qualitative and quantitative data: who their customers are and how they are using the website.

The workshops were also designed to lay a foundation for collaboration, asking teams to consider the experiments other teams were running and whether these insights might be relevant to their products and digital experiences.

These workshops were an interactive learning opportunity between the core team and the supporting teams. And this education was also inspirational: In asking new questions, the different team leads were able to envision entirely new possibilities for how to better develop and position their products leveraging experimentation. They were able to get excited about the possibilities.


Alongside these workshops, the core team was working to document a communications plan. The plan would have two main components:

  1. Clarifying roles and responsibilities as the experimentation program scales, and
  2. Clarifying who needs to be informed and how

To clarify roles and responsibilities, we recommend leveraging RACI, or a similar model. This model helps you map out who owns which piece of the overall task.

RACI Model

R = Responsible: The person who does the work to achieve the task.

A = Accountable: The person who is accountable for the correct and thorough completion of the task.

C = Consulted: The people who provide information for the project and with whom there is two-way communication.

I = Informed: The people kept informed of progress and with whom there is one-way communication.

RACI model example culture change
An example RACI model based on internal activities at WiderFunnel.

Along with clarifying roles and responsibilities, you need a documented communications plan, which should:

  • Identify the information that needs to be communicated
  • List the methods of communication (formal and informal) and how they’ll be utilized
  • Determine the line of communication: Who communicates to whom?
  • Be intentional with timing
Communications plan example culture change
An example communications plan.

MailChimp is an example of an organization that prioritizes communication, and they have seen a lot of success in scaling their experimentation program. I first spoke to the Experimentation Champion there a year ago—she was just getting the program off the ground in the Marketing team. Today, MailChimp is testing on their Marketing site, on 3 of 9 of their product domains, as well as within their technical content team.

The Champion I’m referring to is Lauren Schuman, now Senior Director of Product Insights & Growth. Lauren explained that documentation and communication were a priority from the very beginning at MailChimp.

We documented every single step of the workflow and did a RACI Model associated with it—so that we were very clear on who was doing what, who we needed to consult versus inform. We then mapped the actual communication strategy […] And this has been a major contributor to how we’ve scaled and how we’ll be able to scale in the future.

Lauren Schuman, Senior Director of Product Insights & Growth at MailChimp

Recently, Lauren shared that she now refers to her internal communications plan as a “marketing plan”, adopting a different mentality. She is marketing the experimentation mindset internally, identifying segments, channels, and relevant messages. The intention is to drive enthusiasm, affinity, and adoption.

Getting your organization psyched about experimentation

Inspire, Educate, Inform. These three actions are tools you must use as you scale your experimentation program. You should determine when to create systems that enable these actions at each stage of maturity.

When you assemble your core experimentation team—as the Champion, you have to make sure that you are Inspiring, Educating, and Informing your key stakeholders. Whether this is happening informally or formally, your core team should be excited about experimentation, they should understand it and its value, and the information loop should be closed before you attempt to expand the program into supporting teams.

As you scale, build systems that will help you automate these actions as you pursue the ultimate goal: A culture of experimentation that permeates your entire organization.

Keep in mind that your communications strategy, like experimentation, is an iterative process. Don’t expect perfection right away; work with your core team and, eventually, your supporting teams to determine which messages and channels work best. Be willing to evolve your strategy, but stay committed to Inspiring, Educating, and Informing the people around you.

Are you working to shift organizational culture and promote an experimentation mindset? What challenges are you facing? What successes are you seeing? We’d love to hear from you! Leave your thoughts in the comments section below.


Natasha Wahid

Marketing Manager


Michael St Laurent

Director of Experimentation Strategy & Product Development Lead

Benchmark your experimentation maturity with our new 7-minute maturity assessment and get proven strategies to develop an insight-driving growth machine.

Get started

Source link

Business evolution happens in experimentation sprints: Insights from André Morys, GO Group Digital

Business evolution happens in experimentation sprints: Insights from André Morys, GO Group Digital

Many executives are seeking a “digital transformation” as a lofty solution.

But transformative change really happens in sprints.

That’s because experimentation is the agile approach to business evolution.

When it comes to business evolution—you don’t know, what you don’t know. If you aren’t learning, your business is not evolving at the pace you need to surpass your competitors.

Let’s start with a story…

In the late 1990s, business leaders were hyped on the possibilities the internet offered. Countless start-ups sprouted up to get an early grasp on web business.

But by 2002, when the bubble burst, many companies struggled to survive. Companies like, WorldCom, and WebVan failed completely, and other organizations experienced declining revenues after a period of optimistic growth.

WiderFunnel André Morys konversionsKRAFT
A young, confident André Morys with his business partner at the start of konversionsKRAFT in 1996. (Source: André Morys)

André Morys, Managing Partner of GO Group Digital and Co-Founder of konversionsKRAFT, lived through this experience. In the first five years, it seemed like he had hit pay dirt with his business. His team was confident about their future direction until 2002 when they struggled with declining revenue of 60% over three months.

But when André reflects back on his company’s history, this struggle provided an unprecedented opportunity to learn about leadership and finance, culture and motivation. The aftermath of the bubble accelerated his understanding of how to strategically evolve his business for future growth.

There are many parallels in today’s market. 52% of the Fortune 500 since 2000 don’t exist anymore. And business leaders are constantly battling this threat.

Today, many Executive teams are aspiring to the “digital transformation” solution because how can you keep pace with the market, with the rapid technological change?

WiderFunnel André Morys Unbounce CTA Conference Presentation GO Group Digital
André Morys, Managing Partner of GO Group Digital, presented “The Source of Disruption Is in the Mind of Your Customer” at Unbounce’s 2018 CTA conference.

In this post, you will learn key insights from André Morys, adapted from his presentation, “The Source of Disruption Is in the Mind of the Customer,” at Unbounce’s 2018 CTA conference.

These key insights include:

  • Why we should be focusing on velocity of learnings (not tests) by prioritizing impactful experiments
  • How to speak the same language as your Executive team by pinpointing their emotions and motivations
  • And why a valuable customer experience is at the heart of business evolution.

Scoping out the big picture: The Gartner Hype Cycle

Roy Amara, a researcher, scientist and futurist, claimed that we tend to overestimate the effects of technology in the short term and underestimate the effects in the long term. This phenomenon is called Amara’s Law.

When we think big picture about digital transformation, this forecast is true. We are hyped up on the new tools and technologies when we first adopt them, but once they present challenges, we can become discouraged. Because how can really leverage technology to solve our business problems?

WiderFunnel The Gartner Hype Cycle for Digital Transformation
The Gartner Hype Cycle is a framework for viewing the path from adoption to actually driving business decisions. (Source: Gartner )

The research firm, Gartner, furthered Amara’s Law by introducing the concept of a hype cycle. When it comes to experimentation, WiderFunnel traces the maturity of organizations through its five different stages, including:

  1. The Technology Trigger: You are excited at the possibilities of experimentation but business impact is yet to be proven at this initial stage.
  2. Peak of Inflated Expectations: Early adopters claim success with testing at this stage, but you might be failing to properly leverage experimentation as a strategy.
  3. The Trough of Disillusionment: Initial hype is tapering off. Internal ennthusiasm fades. You might recognize at this stage that experimentation must provide results to continue the investment.
  4. The Slope of Enlightenment: Experimentation is starting to show its possibilities as you understand how to better leverage testing to create business impact.
  5. And the Plateau of Productivity: With consistent bottom-line impacts, you can now start to leverage experimentation as an organizational strategy for business evolution.

Technology can be a solution to business evolution, but leaders need to strategize how to leverage technology to solve real business problems. André articulated that such challenges are actually what is driving your digital transformation.

WiderFunnel André Morys Digital Transformation
When you embrace the pain, you can start to understand the truth to make your business grow, according to André.

As you start to scale your experimentation program, these learnings make your team’s workflow more efficient and allow you to zero in on the hypotheses that can make the most impact.

The good news: You can accelerate your learnings for how to evolve your business through experimentation. Even if these seem small wins, compounded over time, you are truly driving your organization’s growth through digital technology. (For example, what is the calculated impact of a reported 2% lift over 50 experiments? It’s not 100%; it’s a compounded 264%!)

WiderFunnel Digital Transformation throug Experimentation
Experimentation is the agile approach to digital transformation. It facilitates data-driven decision making. (Source: André Morys)

Once organizations introduce a defined process and protocol, have systems and procedures in place for prioritizing experiments by impact, they are able to scale their programs for long-term business evolution.

Relevant resource

Addressing your strategic blind spots: The Dunning-Kruger Effect

It is far more common for people to allow ego to stand in the way of learning.

If you are relying on the HiPPO’s strategy for business evolution, how confident are you in their abilities? And do you think they have the competence to judge their limitations?

When André reflects back on his first five years of business before the bubble burst, he sees how his confidence was an example of the Dunning-Kruger Effect.

The Dunning-Kruger Effect is a cognitive bias where an individual with low ability have mistaken confidence and believe they are more competent than they are.

And people with high competence often view themselves as having lower abilities than in actuality. As André states, “You don’t know, what you don’t know.

WiderFunnel The Dunning-Kruger Effect
The Dunning-Kruger demonstrates how people with low abilities can overestimate their competency, and people with high abilities can underestimate their competency. Hello imposter syndrome!

This cognitive bias has been explored in depth by psychologists David Dunning and Justin Kruger. Their research shows that people that suffer from the Dunning-Kruger Effect may resist constructive criticism if it doesn’t align with their own self-perception. They may question the evaluation and even deem the process as flawed.

Kruger and Dunning’s interpretation is that accurately assessing skill level relies on some of the same core abilities as actually performing that skill, so the least competent suffer a double deficit. Not only are they incompetent, but they lack the mental tools to judge their own incompetence.

When it comes to innovation, the Dunning-Kruger Effect creates a blind spot for threats and opportunities that can affect your business success. Instead, a business leader needs to always interrogate their perception of reality to get closer to the truth.

Truth―more precisely, an accurate understanding of reality―is the essential foundation for producing good outcomes.

André now sees how the growth and learning that came out of the bubble challenged his own self-perception. He began to understand the implications of his business decisions, and he became more in tune with the possibilities of the unknown.

WiderFunnel André Morys konversionsKraft
Today, konversionsKraft has grown to a team of 85. The downturn from the bubble lead to increased learnings in growth, management, culture, finance, leadership, motivation, and more. (Source: André Morys)

Applying his own professional and personal learnings from this experience to the world of experimentation, André sees a chasm between the manager’s aspirations of digital transformation and the optimizer’s experiments that lead to the desired data-driven decision making.

The problem is that [the Dunning-Kruger Effect] happens in organizations all the time,” explains André.

Many optimizers are very skilled in experimentation, they know everything about it: about A/B Testing, confidence levels and statistics, testing tools and psychology. Whereas management has no idea; they don’t get it. They are talking about digital transformation as a big project, while their optimization team is really doing the work that is needed.

What makes André’s argument that organizations need to focus on learnings so compelling is that a culture of experimentation—testing and learning—can really drive your business evolution and lead to the hockey-stick growth that you need to sustain your market.

But the Executive team and the tactical experts need to get on the same page, especially when it comes to successful business evolution.

As an Optimization Champion, you are the catalyst.

Change-agents build bridges between their peers, empowering them to accept change as it comes. They understand how to build and nurture relationships in order to find common ground with others. They are organized and understand how to speak to c-level executives clearly.

André’s comparison of Optimization Champions and Executive teams within an organization with the Dunning-Kruger effect is insightful.

He argued that Optimization Champions have a high-level of competence, but they don’t understand how they can position their work to gain Executive buy-in because they are too immersed into their specialization.

But he also emphasized that Optimization Champions truly are the catalysts of digital transformation. His advice is simple: Optimization Champions need to speak the same language as the Executive team.

Optimizers should stop talking about uplifts and statistics and A/B tests. They should talk about what A/B testing changes within an organization. They should report business impact, not statistical confidence levels, so they are compatible, so they are speaking at the manager’s level.

Experimentation drives the digital transformation at many successful organizations. Just look at Airbnb, or Uber, or Facebook. These organizations test and learn their way to business evolution.

André points out that experimentation facilitates an organization’s digital transformation, but many managers just don’t know it yet.

Your communication of experimentation’s value needs to be accessible to those who aren’t educated in the technical aspects.

And that’s exactly what André recommends. Understand your experimentation program’s internal stakeholders—your Executive team. Understand their fears and anxieties, their emotions and motivations when communicating your experimentation program’s value.

WiderFunnel André Morys Evangelizing experimentation personas of internal stakeholders
André recommends creating personas for your internal stakeholders so you can communicate the value of your experimentation program in a way that they’ll appreciate. (Source: André Morys)

As a marketer, you are best prepared for this task as you can craft your internal stakeholder’s persona, so you can demonstrate—in their language—the impact of your program on the business.

Featured Resource

Build your internal stakeholder persona to get buy-in!

Dive into the emotions and motivations that will resonate with your internal stakeholders to start proving the value of your experimentation program.

In your experimentation sprints, prioritize business impact over speed.

If you’re not failing, you’re not pushing your limits, and if you’re not pushing your limits, you’re not maximizing your potential.

In the world of experimentation, there is an emphasis on experimentation velocity. It makes sense: the more tests that you are running, the more insights you can obtain about your business.

But the buzz around velocity has led many leaders to focus on speed, rather than the quality of insights and the business impact of experiments.

And if you aren’t focusing on business impact, you’ll never get on the same page as your Executive team.

If you decide to test many ideas, quickly, you are sacrificing your ability to really validate and leverage an idea. One winning A/B test may mean quick conversion rate lift, but it doesn’t mean you’ve explored the full potential of that idea.

Experimentation sprints are chock full of business insights and impact—exactly what organizations need to continuously evolve their businesses.

That’s why our emphasis as optimizers should be on velocity of learnings, not just experiments.

It’s an agile approach to business evolution. And that’s a sentiment that was echoed by Johnny Russo, in “The 5 Pillars of Digital Transformation Strategy at Mark’s.”

Because how do we process all this change and learning, without being efficient?” he described.

Those organizations first to adapt will be most prepared. And so I think the foundation has to be an agile methodology.

But optimizers need to ensure they are driving higher impact experiments and deeper learnings by implementing rigorous processes. A prioritization framework ensures you are launching experiments that have the highest potential for results, on the most important areas of your business, with the easiest implementation.

But to increase experiment velocity, you need a defined optimization process.

According to the “State of Experimentation Maturity 2018” original research report, experiment velocity is a focal point for most organizations.

WiderFunnel State of Experimentation Maturity Velocity Increase
The majority of both Small and Medium Enterprises (52%) and Large Enterprises (64%) plan to increase experiment velocity in the next 12 months.

52% of Small and Medium Enterprises and 64% of Large Enterprises in the survey indicated they plan to increase experiment velocity in the next year.

However, only 24% and 23% (respectively) of these organizations plan to increase budget, which can only add emphasis on workflow efficiency and prioritization so that you are not straining your resources.

One of the most common roadblocks to increasing velocity is workflow efficiency. Review and document your workflows from ideation to analysis to ensure seamless experiment execution,” explains Natasha Wahid in the research report.

Another common roadblock is a lack of resources, particularly in Design and Web Development. Ensure you have the right team in place to up your velocity, and plan for possible bottlenecks.

As André articulated, focusing on business impact instead of speed will ensure you are learning faster than your competition.

That’s because digital transformation is not about implementing technology, it’s about leveraging technology to accelerate your business.

How to use the PIE Prioritization Framework to identify the most impactful experiments.

You can’t test everywhere or everything at once. With limited time and resources and, most importantly, limited traffic to allocate to each test, prioritization is a key part of your experimentation plan.

Prioritizing where you invest energy will give you better returns by emphasizing pages that are more important to your business.

The PIE Framework is made up of the three criteria you should consider to prioritize which pages to test and in which order: Potential, Importance, and Ease.

WiderFunnel PIE Prioritization Framework
The PIE Prioritization framework allows you to zero in on those experiments that can drive the most business impact. That’s how you get executive-level buy-in!

Potential: How much improvement can be made on this page(s)? You should prioritize your worst performers. This should take into account your web analytics data, customer data, and expert heuristic analysis of user scenarios.

Importance: How valuable is the traffic to this page(s)? Your most important pages are those with the highest volume and the costliest traffic. You may have identified pages that perform terribly, but if they don’t have significant volume of costly traffic, they aren’t experimentation priorities.

Ease: How difficult will it be to implement an experiment on a page or template? The final consideration is the degree of difficulty of actually running a test on this page, which includes technical implementation, and organizational or political barriers.

The less time and resources you need to invest for the same return, the better. This includes both technical and “political” ease. A page that would be technically easy to test on may have many stakeholders or vested interests that can cause barriers (like your homepage, for example).

You can quantify each of your potential opportunities based on these criteria to create your test priority list. We use the PIE Framework in a table to turn all of these data inputs into an objective number ranking.

Learn more about PIE

Seek the “truth” in a delightful customer experience.

Imagine your a taxi service. And you are seeing Uber’s market success as a threat to your livelihood.

What makes them more successful?

Get out the whiteboards and some might write: “We need an app!” or “We need to hire a data scientist!” Because on the superficial level, data and technology seem pivotal to a digital transformation.

But when you look deeper at Uber’s strategy, you will see that they focus on delighting their customers with their experience.

And because of Uber’s success, how people get rides has radically changed.

WiderFunnel Uber Growth Customer Experience
Look at the hockey-stick growth of Uber. It’s all attributed to a valuable customer experience. (Source: )

But Uber is not the only example.

Many businesses have transformed the market through delightful customer experiences.

Amazon makes online ordering a breeze. No more long wait times, André joked that he has to slow down their service so that he will actually be there to accept the order.

And Airbnb—they’ve made vacationing a unique and desirable experience.

André, an expert in emotional marketing and the Limbic Model, emphasized the need to go beyond conversions and focus on a customer experience that delights.

Companies who fail to embrace CX as strategic path to growth won’t just be lagging, they’ll get left behind.

And that’s because you can drive experiments with the most business impact by honing in on your customer experience.

That means, you have to understand your customer—their fears and anxieties, their thoughts and desires—when designing an experience to meet their emotional needs and states.

The most successful organizations have honed in on what makes their experience delightful. In “Moving the needle: Strategic metric setting for your experimentation program,” I talked about creating a True North metric that align to your value proposition, as a way of creating internal focus on your customer experience.

Slack, a team collaboration program focuses on optimizing for teams that send over 2,000 messages.

LinkedIn focuses on quality sign-up, ensuring that new users are actively making connections on the social platform.

And Facebook optimizes for new users who make 10 friends in seven days.

See it’s not tools and technologies that evolve these businesses. It’s how they leverage digital technologies across the organization to solve real business problems.

And when you align your customer experience goals with your experimentation program, you are the competition.

You learn and adapt with each new experiment to what your customer wants and needs from your experience. That’s how you drive business impact. That’s how you get internal buy-in.

But what’s more—that’s how you stay relevant in the market.

And if you are experimenting, keep slogging through the “trough of disillusionment”.

André Morys knows from experience that business that do, can reach enlightenment.

What are your biggest challenges with experimentation that can be attributed to the “trough of disillusionment”? We’d love to hear your comments!


Lindsay Kwan

Marketing Communications Specialist

In this roadmap for the executive, we explore the biggest challenges faced by enterprise organizations as they work to embed experimentation within their infrastructure – and how to surmount them.

Get your roadmap

Source link

Experimentation in product development: How you can maximize the customer experience

Experimentation in product development: How you can maximize the customer experience

Are you a product builder or a maximizer?

Hila Qu, Vice President of Growth at Acorns, has a theory.

She says there are two kinds of product managers: the builders and the maximizers.

Ideally, on your product team, you have both: the builders who conceptualize a solution in product development, from the ideation stage all the way until the customer roll-out.

WiderFunnel Product Teams as Builders
Product Builders are essential to organizations because they conceive the product that provides value to the customer in the first place.

They’re very important because, without them, there’s nothing; they are building the value from zero to one,” Hila explains.

And then, there are those that can maximize the customer’s value through experimentation in product development.

The bulk of the product value is already created, but how can you maximize that value? The maximizers help customers understand the product better when a new product or feature is launched because we often assume people will know how to use it.

Hila Qu

VP of Growth at Acorns

WiderFunnel Product Teams Maximing the Value of Customer Experience
Product Maximizers build upon the value of the product through experimentation.

It’s an interesting distinction—between developing and experimentation.

Product development is all about making your product solve your customer’s real-world problem, and the builders aim to make the process frictionless so they will create a habit out of returning.

But product development teams might not know if they were successful. Or how they could improve the experience for the users.

Experimentation enables the maximizers to improve upon the value that is already developed, to make the customer experience more delightful.

Actually, your product team can leverage experimentation where ever they are in the product development cycle: They can test an idea before launch; Or they can improve your feature after it’s rolled out to your customers.

Experimentation is iterative; your product team learns and develops as they gain insights about your business and your customers.

If you are exploring the possibilities of product experimentation at your organization; if you are a marketing leader looking to evangelize experimentation throughout the business—this post is for you.

In this post, we’re tackling the essentials of experimentation in product development. Essentials like:

  • The breadth and depth of experimentation in product development so you can start dreaming up the possibilities.
  • An understanding of the stakes when experimenting within the product development cycle, especially when those hard-earned customers are interacting with your brand.
  • And why experimentation in product development is a leap toward a customer-centric organization, a strategy for growth.

But we also wanted to show you examples. So, we talked to product managers at Optimizely, Acorns, RVShare and Stitch Fix about the opportunities for product experimentation in reaching business goals.

And they are clearly seeing results.

You might be thinking: what is product experimentation?

Client-side experimentation is based on the premise that what-you-see-is-what-you-get. It’s about visual changes to the content hierarchy of the website.

Innovative marketers have been doing this form of optimization for years, and they’ve seen the success. So what’s next?

Product experimentation is often top of mind when it comes to expanding an optimization program internally. That’s because there are more possibilities.

The product life cycle is usually more elongated than the marketing lifecycle,” explains Giannis Psaroudakis, Director, Product at Optimizely.

In the marketing lifecycle, you have a campaign and you have a very specific goal, which is sometimes short-lived – you want to increase your leads, for instance.

If you are optimizing your website to get a customer to convert, only to have a product that does not live up to the customer’s expectations—those hard-to-win customers are going to look elsewhere.

But it can be tricky.

Product managers don’t always get it right the first time, and that’s why it’s important to have a feedback loop with your customers.

It’s important to maximize the value of your product. Understanding how your customers are finding value and optimizing their journey so that they can find that value faster.

“Even well-researched products can suffer due to the gap between what customers think they want and what their behaviors reveal they actually want.

By testing a new feature or even a variation of that feature, your product team can see if it is improving the customer experience, or actually making it worse.

The experimentation mindset aligned with the premise of a minimum viable product, seeing how your customers react through a gradual or even incremental roll-out. It allows you to test new ideas before a full launch.

Don’t frame it as a product launch. Just frame it as an experiment.

Dan Siroker

Optimizely Co-Founder and Executive Chairman

If you crave that buzz of a big launch, experimentation after-the-fact enables you to get the most out of those ideas—and your organization’s investment.

By the time something this big has been built, the launch is very, very unlikely to be permanently rolled back no matter what the metrics say.

Rather, the randomized experiment, in this case, is for visibility, and to provide information that might help with making future decisions.

Announcing a feature or product through an official launch can inspire adoption—particularly if you have a large customer base to which you can communicate.

But once your product is rolled out, you can start to understand how your customers are interacting with the features at every touchpoint through experimentation.

Product experimentation should make the job of the end user easier or more helpful. If that’s not the goal in your company than you might be running it the wrong way.

Martijn Scheijbeler

Vice President of Marketing at RVShare

Those learnings can be brought forth into future product evolutions—whether that’s changing copy to highlight your value proposition, or smoothing out the funnel flow to reduce friction.

And you can even experiment after your customer logs into your experience, especially when confidentiality is essential for your business.

Do what your customer wants: A painted door experiment example from Optimizely

We created a painted door experiment informing customers we intend to give you this feature, sign up to get early access. We wanted to test if its value proposition resonated with our users,” explains Giannis Psaroudakis of Optimizely.

For those that don’t know–a painted door or a fake door experiment is a way of gauging customer interest in a feature, service or product without building anything.

When the customer clicks a call-to-action button or a link, or even registers with their information, they are notified that it has yet to exist. And they might see something like:

Hey! We haven’t actually built this feature yet, but are you interested?

Of course, you have to know if your customers would be open to this type of experience. You would have to make the experience intriguing enough so that it doesn’t cause frustration.

But you get the data to see if customers find that appealing. Or at least made them curious enough to click or register. And for Optimizely, this worked:

We had several customers enthusiastically send us feedback. It was overwhelming—we received dozens and dozens of requests which is quite uncommon and we even had a customer send us a photo of a thumbs up.

WiderFunnel Experimentation in Product Development Optimizely painted door experiment example
Experimentation is a feedback loop with your customers, but in Optimizely’s painted door experiment, they were able to delight their users with the hope of a new feature.

A painted door experiment, like this, truly exemplifies the power of running a cheap form of experiment as a gauge for the next step.

It gives you the confidence that you’re moving in the right direction, without a single line of code or effort by an engineer.

But more importantly, it’s that feedback loop with your customers.

Relevant resources

The scientific method of product experimentation

Let me first clarify: you don’t test features in product experimentation, you test hypotheses.

Product experimentation takes an optimizer’s brain: It takes an analytical mind to be able to see the opportunities for product experimentation. It also needs the scientific method to turn ideas into measurable hypotheses.

Move away from the mental model of thinking of new products and new features you want to build, in the form of a list of requirements, and instead, think of them as hypotheses.

Giannis Psaroudakis

Director, Product at Optimizely

Rather than saying, “we need to build feature X that has this requirement and that requirement, because customers asked us to”, you start with presenting these same requirements in the form of if/then statements.

This is so important because it sidesteps any HiPPO (highest paid person’s opinion), and it persuades you to think about your product—how you’re building the product—in the form of experiments. In other words, hypotheses that you can validate with data.

Making data-driven decisions is important in product experimentation because the stakes are higher.

You are experimenting within your product experience on your many already acquired customers so you need to have high quality assurance and more guard rails to what makes it to the experimentation stage.

We have a more rigorous review process for product experiment hypotheses,” recalls Giannis Psaroudakis.

We have an experiment review, which is more of an advisory process that allows us to surface new product experimentation ideas and specifically review our hypotheses.

Anyone in the organization can submit their “experiment briefs”—customer challenge, hypotheses, and metrics of success—through Optimizely’s Program Management platform.

And we revise these experiment briefs in the weekly experiment reviews to make sure that whoever is planning to run a product experiment has carefully thought about the hypothesis and the metrics of success.

The scientific method ensures that you can make the data-driven decisions on how your product—and the customer experience—should evolve.

Free Worksheet

Identify opportunities for maximizing your product’s value.

When you use the scientific method in experimentation, you don’t test features; you test hypotheses. Start planning your first product experiment with our hypothesis worksheet.

Digging deep into data to find that first product experiment opportunity at Acorns

When I joined two years ago, my goal was to improve customer retention,” recalls Hila Qu. “But it is a very broad goal: What do we mean by improving customer retention?

Hila described how she talked to her co-workers—those stakeholders who have been exploring the problem. She asked them the reasons people left their app and their opinions as to what might improve customer engagement.

But she didn’t stop there.

I also worked with our data analytics team to do a quantitative analysis. Basically, I was trying to identify which customer behavior encouraged people to stay.

For example, I looked at the retention curve after a customer completed action A in the product, and compared that to the retention curve of people who didn’t complete action A in the product within their first month.

Using that methodology, I was able to compare different behaviors, because for any digital product, there are a lot of actions user can take.

I was able to narrow down to one particular user behavior where I saw that if the user completes that action, their retention is much better that if the user doesn’t. I also saw that many customers weren’t actually completing that action.

Through her analysis, Hila identified that if a new customer completed action A at the onset of their journey, it had an influence on the customer’s future behavior.

It had a much bigger impact because at that moment, we had people’s attention and they were excited. They wanted to get started with the product.

Hila then mapped out the journey, considering the many paths that a customer could take to complete action A. And there were several:

“For example, you can go to the menu, click that menu item and use that feature; Or, you can basically send them an email with that call-to-action to use that feature. I identified all of the paths so I could understand which one I could focus on first.

So, my hypothesis was based on the data; it seems like if we can get more people to complete action A, the retention will be better. I narrowed down to new customers and if I could get more of them to complete action A, it would be most impactful and easier to influence people’s behavior.

Her first experiment was simple. She didn’t have the development or engineering resources since she was only initiating a product experimentation program.

She hadn’t yet proven experimentation’s value internally, so she needed to identify an opportunity where she could show business impact from a small investment of resources.

We already had a modal in our registration flow that asked people to take action A, but the modal was at the very end of the flow. And you only see the modal if you completed another action. So, I narrowed down to this particular area.

I wanted to test copy. That experiment had the potential for a very high ROI. If it worked, it could have a big impact, but also the effort required was relatively small. It was really just changing the copy and testing different copy against each other.

And it worked.

We launched it and saw over 60% improvement in conversion rate.

Simple or complex—product experimentation can have a big impact.

Further resources

Experimenting broadly and deeply in product development

When it comes to experimentation in product development, you have more opportunity to optimize your experience. That’s because you can experiment broadly and deeply into the product development cycle.

WiderFunnel Experimentation in Product Development Cycle

There are plenty of opportunities to experiment in the product development cycle.

Share the insight:

Consider the stages of product development:

First, you have the Discovery stage where you ideate new features or products and this is particularly relevant when you ideate in the form of hypotheses.

According to Meg Watson, Product Manager at Stitch Fix, her team also conducts an analysis to determine if they should proceed into product development.

Next, you move into the Design stage, according to Giannis Psaroudakis, where you can evaluate your hypotheses through early user studies or rapid prototyping. This prototyping is where experimentation begins in terms of usability.

Then, you go on to the Development stage where you dive into coding or testing your actual code.

And afterward, you have the Roll-out stage, when you make sure your hypotheses still stand after launch and there’s nothing negatively affecting the quality of the final product or the customer experience.

In traditional product development, this is where you might stop. Your ideas might win, but they also might fail. And there’s no defined evaluation for where you can improve.

That’s why in product experimentation, you also have the Maximizing stage, that Hila Qu described: What can be improved? What can we evolve? What can make the customer experience frictionless, more delightful?

When I’ve looked at some of the things we’ve started with and usability testing, and then compare that to the product that ships, X months later, a lot happens in between those stages. We start with sending a prototype through usability and then we have all the different optimization tests that run after something goes live.” confirms Meg Watson.

Throughout the product development cycle, with the experimentation mindset, you are constantly exploring and validating the possibilities for new features or products by generating that feedback loop with your customers.

But perhaps more importantly, experimentation in product development means you can test deep in your stack.

Unlike client-side experimentation, with its what-you-see-is-what-you-get approach, you can test what you can’t see. Product experimentation incorporates server-side testing:

You can test changes on your server side and improve the performance of your back-end systems,” Giannis Psaroudakis affirms.

If you’re running any machine learning models that give recommendations to customers, you can experiment with those, for instance.

In client-side experimentation, a variation is run through the web browser or mobile app using a visual editor and a single JavaScript line. Once the variation is proven, you can build it out by implementing a code pack.

But in server-side experimentation, the variation is coded on servers. So, it can be more resource intensive because you build up-front.

WiderFunnel Experimentation in product development client-side server-side
You can see that server-side experimentation requires that you build upfront. (Source: “Why Experiment Server-Side?” by Optimizely)

That way, you can experiment with how a product functions, like the machine learning algorithms that enables you to deliver a personalized experience.

And that’s why product experimentation has so much potential.

Learn more from these innovative organizations

When it comes to experimentation, it’s all about the customer experience.

Ultimately the goal of all this, the reason we’re doing this, is to give the customer a better experience with Stitch Fix and to make sure that it’s effortless. It’s fun. it’s delightful. And that they truly have a good experience with all of our products.

Experimentation in product development allows you to get deeper into the stack, deeper into your customer’s experience so that you are delivering the best possible solution and a delightful one at that.

For many organizations, it’s an untapped source of growth.

The gap between product and marketing teams is becoming smaller and smaller, and from a customer experience perspective, it’s blended.

The customer experience has to be consistent and compelling, and work seamlessly between the pieces that are controlled by the two teams, in order to have the customer finish that journey from a random visit or from your ads, to the point where they use the product for the first time and they know the value, they see the benefit.

WiderFunnel Product and Marketing Customer Experience
If the customer doesn’t view marketing and product as separate parts of their journey, organizations shouldn’t either.

But we need to stop siloing product and marketing.

With an experimentation mindset, organizations need to unite internally to spread those insights throughout every customer touchpoint.

That means experimenting constantly—both client-side and server-side—and sharing the insights across both marketing and product development teams.

Because your customers don’t see your landing page or your product funnel as distinct experiences. Instead, you need to focus on the entire customer journey.

What are your burning product experimentation questions? We’d love to hear them below.


Lindsay Kwan

Marketing Communications Specialist

Benchmark your experimentation maturity with our new 7-minute maturity assessment and get proven strategies to develop an insight-driving growth machine.

Get started

Source link

The future of the digital customer experience: 6 experimentation trends for disruptive businesses in 2019

The future of the digital customer experience: 6 experimentation trends for disruptive businesses in 2019

You are the catalyst for future growth.

This past September, hundreds of specialists at organizations across industry verticals flocked to #Opticon18, the largest experimentation conference in North America; and it was clear; the driving force is you—the Optimization Champion.

Experimentation as a strategy for future business evolution and innovation is happening now.

The most successful organizations are experimenting (or at least planning to!) across all customer touchpoints. Because customer experience is king.

To keep pace, you’ll need to stay on the pulse of what’s happening with experimentation: new tech and tool developments, the latest strategies for scaling experimentation, and the emerging trends that will define your business in the future.

Because your ability to adapt will lay the foundation for the radical change that is set to happen in the business world:

The next 10 years will generate an order of magnitude more change than we have seen in the last 10 years.

Brian Hopkins, Ted Schadler, and James McCormick

In 2019, you will want to accelerate your strategy with these six experimentation trends that will pave the future of your digital customer experience:

Trend 1: The currency of business insights

Customer data, machine learning algorithms, the latest technology stack—no technological development is worth implementing if you can’t leverage that data into business insights.

Business insights are the valuable intel that allows you to experiment and evolve. To innovate and proliferate your learnings across your entire organization.

Data is the red blood of an insights-driven business—there can never be enough flowing in the veins. Look always to tap more—and more relevant—data.

Brian Hopkins, Ted Schadler, and James McCormick

Your ability to accelerate the speed and transmission of insights across different teams is the currency of the future. Business insights will underpin the radical change we will see in the next 10 years.

Data analytics and software enable insights-driven organizations to sift through an immense amount of information to glean transferable insights.

Because this trend hinges on human intelligence: the ability to accumulate a large quantity of quality data sources and to be able to glean actionable insights. Insights that reveal something about your systems and processes, your product or services, and especially your customer:

Insights-driven businesses bring insight, not just data, into every decision, and they know exactly how to use them for greatest advantage across the entire customer life cycle. For these firms, digital insights and what they do with them are their secret weapons to disrupt your market and steal your customers.

Brian Hopkins, Ted Schadler, and James McCormick

But more importantly, the most successful organizations will find ways to close the loop—bring insights forward and experiment with them at different touchpoints.

Insights-Driven Experimentation

The flow of insights drives your experimentation program, maximizing your organizational learning capacity.

Share the insight:

You must democratize your data and insights so anyone in the organization can harness them for an improved customer experience.

At Uber, for instance, 50% of their employees have access to an insights database which helps to inform their decision making on a daily basis, according to the Forrester report.

Because information is power.

There are many things that you need to get right to create internal alignment and scale insights across the enterprise. However the most important is having the executive team demanding this approach and a top-down strategy guiding the synchronization of teams around a common practice.

But this is a challenge for any traditional, non-agile organization.

In fact, Brian Hopkins, Ted Schadler and James McCormick predict that insights-driven businesses will grow eight times faster than the projected 3.5% global GDP Growth. More granularly, their predictions include that insights-driven public companies will grow 27% annually and startups will grow 40%.

But this might not be possible without the top-level support of changed processes and systems.

What processes and protocol can you document to ensure that business insights are spread throughout your organization?

A Marketer’s take on this trend

The dissemination of insights is crucial if you really want to move your organization forward. The learnings you generate from experimentation can’t live in a single team – particularly those insights about your customer’s emotional states and contexts. Because these insights can most likely be applied and tested at many touch points throughout your business.

Now is the time to figure out what systems you need in place to ensure the right people have access to insights from data, experimentation, and customer research. This may be as simple as an experiment insights archive, or it may require a more intentional dissemination effort.

Natasha Wahid

Marketing Lead

Resources to get you started…

Trend 2: The cross-organizational experimentation mindset

Thomas Edison would be thrilled to be alive today, if he could see the stuff that is really going on, the stuff that you are all doing.

It’s not a surprise that Forrester analysts predict unprecedented growth for insights-driven businesses in the next decade. Experimentation refines ideas into validated insights, evolving the digital customer experience.

And with more organizations adopting the experimentation mindset of testing and learning across every department, the ability to generate and leverage business insights increases exponentially.

In the next 10 years, we will see widespread adoption of experimentation across all touch points to validate all marketing activity and focus our limited time and resources on the high-impact areas.

Nick So

Director of Strategy

The future of the digital customer experience is through experimentation. In the next 10 years, you will see more and more organizations experimenting across every customer touchpoint, in order to optimize their entire journey.

Your first step is to de-silo experimentation in your organization. Instead of relegating experimentation as a side-strategy, organizations will need to implement the structures and processes to enable experimentation in every team.

To become a true experimentation organization, you need scale and scope. Scale is about running many experiments and scope is about getting all groups across an organization to participate in experiments.

The most successful organizations are already on board with cross-organizational experimentation. According to Stefan Thomke, at organizations like P&G, Uber, Airbnb, or Bing, experimentation is going on at all times.

[At Bing, at] any point in time, there’s billions and trillions of variations,” explains Stefan Thomke. “And by the way, the success rate at Bing, alone, is only 10-20% of what they try.

WiderFunnel High Velocity Experimentation Examples

Real-world examples of experimentation at scale

The most successful organizations are experimenting at a high velocity, gathering insights from both winning and losing variations.

Share the insight:

And despite a low win-rate, these organizations are investing in experimentation as their cross-organizational strategy. Because it’s not about winning or losing — that’s thinking too small, too immediate.

That’s because experimentation competency across their organization is their competitive advantage. They are testing large-scale and high-velocity because they involve every team, every department. Experimentation is all about gathering business insights.

At its peak maturity, experimentation is a cultural mindset that spans across organizational departments, marketing channels, and throughout executive management.

So where do you get started? We’ve got it figured out.

The 5 pillars of an effective experimentation program

Cross-organization experimentation requires a scaling strategy. It requires focused intention, a multi-pronged approach to your process, your metrics, your culture, your expertise, and your tech stack.

Based on years of analysis of experimentation programs, and through surveying Optimization Champions at organizations all over North America in “State of Experimentation Maturity 2018” report, we’ve identified what makes the most successful programs gain traction across departments, across activities.

And it’s called the PACET framework.

The PACET Framework

WiderFunnel PACET framework for scaling experimentation

WiderFunnel’s PACET Framework

By focusing on the five core pillars of process, accountability, culture, expertise, and technology, you can scale and mature your experimentation program.

Share the insight:

Our findings informed WiderFunnel’s framework: PACET. And it includes these five pillars:


This pillar includes an organization’s experimentation protocol and methodology, process for ideation and prioritization, experiment design, and measurement of success.


The most mature organizations keep process and accountability at the core of their experimentation strategy, fuelling how experiments are developed, and results are analyzed, understood, and leveraged.


Culture is crucial when defining experimentation maturity: Does your organization celebrate testing and learning? Are people encouraged to try (and fail) and try again?

This pillar includes organizational buy-in for experimentation, program support from the C-level, and cross-team participation in an experimentation program.


An experimentation program needs expertise and resources. The amount of time and full-time team members dedicated to experimentation is reflective of an organization’s maturity.

This pillar includes people and skill sets: strategists, analysts, designers, developers, project managers, product owners, third-party partners, as well as hours dedicated to experimentation.


Experimentation maturity requires a well-rounded technology stack. Experimentation and personalization tools, visitor engagement tools, customer data tools, project management tools. Mature organizations have the right tools in place to ensure they can develop the best possible hypotheses and have reliable data.

Your first step is to evaluate how developed each of these core pillars are within your organization, so you can set your sights on your future growth.

Trend 3: Empowered product experimentation

Experimentation has become the product. Your product is the culmination of user feedback and quantitative data tied to your business goals. And experimentation is the engine that brings it all together to validate the way forward.

Just as Stefan Thomke mentioned, your experimentation program’s scale and scope are essential for driving your future growth. If you are considering how to grow your program, empowered product experimentation should be your next step.

There are numerous untapped opportunities:

Server-side experimentation has really opened up what is possible with product experimentation. Allowing development teams to build experimentation directly into their sprints and workflows,” clarifies Thomas Davis, Senior Web Developer at WiderFunnel.

WiderFunnel Experimentation in Product Development Cycle

The product lifecycle

You can experiment deeper into your stack with product experimentation, including with machine learning algorithms, log-in states, and more.

Share the insight:

But besides the ability to experiment throughout the development lifecycle, you also have the opportunity to maximize your digital customer experience by building off the value that is already created.

Successful product managers create an experience that delights, an experience that meets the customer’s emotional needs and states in the context of your product. And continuous and iterative experimentation makes certain that you are moving toward this end goal.

You can heighten the positive emotions that your customer experiences, and minimize the friction points to make it more sticky.

And that experimentation mindset will be critical to ensuring the longevity of your product in the marketplace.

A Developer’s take on the trend

Nothing is more frustrating than building out a fully integrated feature that has a negative effect on the business. Product experimentation stops developers wasting time building out fully polished features that will just be rolled back.

Thomas Davis

Senior Web Developer

Trend 4: The evolution of the Digital Experience Stack

Delivering exceptional customer experiences at scale is high-pressure for the disruptive business leader. It’s a fast-paced market and they know they have to keep up.

Marketing is a ‘jack-of-all-trades’ discipline,” explains Sergiy Bondarenko, Marketing Operations Analyst at WiderFunnel.

Marketers have to be experts in copywriting, sales, design, psychology, web technology, app technology, SEO, paid traffic acquisition and demand generation, social media, public relations, etc.

Naturally, there isn’t a single person who can excel in all these disciplines. This is where many tool vendors come in, promising to ‘fill gaps’, and marketers fall into the trap of thinking a tool can replace skills.

When it comes to technology, gone are the days of finding that traditional one-tool solution. These legacy suites evolve slowly, delivering mediocre results across the board. At that pace, how could you ever stand out amongst your competitors?

Insights-driven businesses are 137% more likely to differentiate with data and analytics.

But implementing new tools and technologies without an overarching martech strategy will lead to poor results as well.

Since 2007, we have gone from ~150 martech vendors to over ~7,000 in 2018. Unsurprisingly, marketers are now suffering from the ‘shiny object’ syndrome—every new tool promises to solve every problem there is, and if it’s trendy, then it’s almost an obligation to work it into the existing workflow—or risk being seen as a laggard.

Sergiy Bondarenko

Marketing Operations Analyst

The new Digital Experience Stack is an innovative solution, particularly for those disruptive businesses that want the best of the tech worlds.

WiderFunnel The Digital Experience Stack DXS
Some of the best-in-class tools have partnered to create The Digital Experience Stack (DXS). Source: Optimizely.

And it ensures you have the well-rounded technology stack to empower your organization’s experimentation at scale.

If you have 100 tools that are fragmented, don’t have an open API, and are sparsely used, then you have a problem on your hands,” states Sergiy Bondarenko.

Firstly, team productivity and happiness will suffer. This will eventually trickle down into underperforming operational metrics and will have a negative impact on the KPIs.

A bloated martech stack also means a bloated budget, and you never want to have a bloated marketing budget—it creates tension and a lack of trust with organizational leaders like the CEO and CFO.

And no organization wants that.

An Experimentation Strategist’s take on the trend

The digital experience stack is a great approach that enables businesses to work with a diverse range of best-in-class technologies, and enables technology companies to continue to focus on their area of expertise. Collaboration, not competition, to better support the industry as a whole.

James Flory

Senior Experimentation Strategist

Trend 5: The re-framing of personalization

Experimentation ensures that businesses are innovating and evolving. But, it doesn’t mean that it is a separate strategy. It is the underpinning methodology of getting any and every strategy right.

Personalization is just one technique within the methodology of experimentation.

It’s not one or the other.

So, your experimentation team shouldn’t be siloed from your personalization efforts.

We’ve seen a lot of hype around personalization in recent years, but many organizations are only aspiring to the level where they can deliver individualized experiences to their customers. That’s the 1:1 experiences that many tools claim to provide.

WiderFunnel Marketing Personalization
Providing a more customized user experience often starts with segmenting your audience, but any personalization tactic needs to be validated through experimentation.

But, as Mike St. Laurent, Director of Experimentation Strategy and Product Development Lead points out: “Most companies do not have the necessary data collection and segmentation capabilities in place to even be thinking about personalization as a strategy.

2019 will be the year of laying the technical groundwork so that companies have the tools they need to test relevant customer experiences effectively.

Mike St. Laurent

Director of Experimentation Strategy and Product Development Lead

You also need to keep in mind that any tactic needs to be proven; not every implementation of personalization will deliver results.

If you have an idea on how to leverage personalization in your strategy, validate your hypotheses through experimentation.

The end goal is to create digital experiences that are highly relevant to the customer in your business context. But you should only want that as a means of generating a higher customer lifetime value.

An Experimentation Strategist’s take on the trend

Creating relevant experiences can be an effective way to improve conversions, but companies are realizing they shouldn’t be personalizing just to say they are doing it.

Companies are starting to understand that just because something is “personalized” doesn’t mean it’s more effective. A personalized experience needs to be tested the same as any other change to a digital experience.

Mike St. Laurent

Director of Experimentation Strategy and Product Development Lead

Get well-versed on this topic…

Trend 6: True customer empathy

Businesses have long been trying to solve their customer’s pain points. But what has been missing from the conversation is true customer empathy.

Because you don’t want your customer to only have their problem solved. You want them to feel an affiliation with your brand and with your experience. You want them to be delighted.

widerfunnel customer delight
Are you creating delight for the individuals who are your customers?

True customer empathy means understanding your customer’s full spectrum of emotions within your experience: knowing what emotions they feel when their expectations are met and how they feel when their expectations are not met.

In the digital world, customers can access your brand on many touch points: social media, email newsletters, your website. All of which offers plenty of opportunities to connect with your customers.

Where are the points of friction and where are the points of delight in your experience?

Unfortunately, at least one unintended bad customer experience is part and parcel of any new launch; companies simply can’t predetermine how every part of their customers’ experience is impacted by design or development decisions made during the feature development process.

A crucial post-launch practice at FullStory is something we call ‘game film‘—a process where we auto-play sessions of users interacting with the new feature and note down how many bad experiences they encounter.

Whether through game film or some other practice, the point is that everyone should have a built-in mechanism to monitor these empathy-inducing moments of frustration for customers.

Jordan Woods

Marketing at FullStory

True customer empathy leads to an understanding of how you can maximize and minimize the feelings your customer experiences at these different points within your experience, so that your brand can align more closely with your customer’s emotional needs and states.

In 2005, when Bain & Company surveyed 362 firms, 80% of companies stated that they were customer centric. That sounds promising until you consider their customers’ response: Only 8% of customers agreed.

Clearly, there is a disconnect.

So, how can you get deeper than demographics to not only understand your customer, but to anticipate their emotional response? How can your organization become genuinely empathetic to their customers?

Start by listening to your customers at every touchpoint.

Marketing Trends Customer Journey
The customer experience is a holistic journey across multiple touchpoints. Having empathy for your customer’s emotional needs and states is crucial for making their experience delightful.

Live chat. Social media listening. Customer surveys. These methods are a starting point. But true customer empathy only comes from deep inquiry and the thick data that results:

Research techniques — such as contextual inquiry, diary studies, ethnographic research and others — can generate thick data that allows you to understand your customer’s emotional needs.

True customer empathy is also a rich source of hypothesis ideation. You can validate this deep understanding of your customer through experimentation to see if your hypotheses stand true.

A UX Researcher’s take on the trend

People will always be the centre of any business. Understanding those people — your users — and their circumstances will help you generate powerful hypotheses. But the key is to take these insights forward through each of your experiments to drive and scale a sophisticated experimentation program.

Kim Quach

UX Research Specialist

Remember what you do now counts.

Your leadership, your strategies, your experiments are driving your organization into the future. What you do now accelerates the growth of your company.

It takes just one person to lead the change. The more you embrace the trends and technologies of the future, the more ready you are to embrace the pace of change.

But know that you don’t have to bear the burden alone.

Build the right insights partnerships – don’t go it alone. You probably won’t own all the data, expertise, or technology. We expect most companies to work with a wide variety of insights services partners.

Brian Hopkins, Ted Schadler, and James McCormick

You can still lead the charge.

As the more determined you are to push the boundaries of how your organization operates, the more likely you can evolve with the rapid growth that your organization can facilitate through experimentation.

But it’s not just about you and your organization—it’s about your organization’s purpose, your vision—the reason why behind your work.

And that is your customers.

Because a delightful digital experience is how your brand stays relevant—now and in the next decade.

What trends stand out as most important to your future growth? Let’s start a conversation in the comment section below.


Lindsay Kwan

Marketing Communications Specialist

Benchmark your experimentation maturity with our new 7-minute maturity assessment and get proven strategies to develop an insight-driving growth machine.

Get started

Source link

Year in review: Our top 9 experimentation articles and news of 2018

Year in review: Our top 9 experimentation articles and news of 2018

2018 was a year of rapid change for many organizations.

That’s because experimentation is becoming an essential business practice.

Over the course of the year, we witnessed organizations go from stage one to stage three on our experimentation maturity scale in just a few short months.

Once they had buy-in from the top and a will to galvanize support across departments, they were able to introduce experimentation to multiple departments.

But even if you just got started with experimentation this year, surpassing the inertia and running your first few experiments has already transformed your business.

And because we are in the business of experimentation, because testing is our livelihood, we’ve also had an incredible year.

Looking back over the past twelve months, I’ve compiled our top nine experimentation articles and news:

1. Matt Wright’s “How Thick Data Helps You Build Emotional Connections With Customers” on CMSWire

This past June, Matt authored the CMSWire article, “How Thick Data Helps You Build Emotional Connections with Customers,” to dive into the process of generating thick data about what makes your customers tick.

WiderFunnel experimentation articles thick data
Matt Wright, Head of UX Research, dove into how to leverage your thick data to create emotional connections with your customers.

In today’s data-driven world, marketers amass immense amounts of customer information through numerous sources such as analytics, CRMs and loyalty programs — all of which provide plenty of quantitative data about customers. This type of data offers the when, where, what and how of your customers’ interactions with your experiences. But critically, it does not provide the why.

Matt Wright

Head of UX Research

Matt’s thought leadership in this article provides a glimpse into his work on WiderFunnel’s MotivationLab, the deep-dive research that provides insights about customers’ emotional needs and states for more impactful experimentation programs.

But what’s more, he shows you the secret to creating delightful customer experiences that increase wallet share and loyalty for years to come.

2. Experimentation in product development: How to maximize the customer experience

Innovative marketers have been doing website optimization for years. So what’s next?

Product experimentation is top-of-mind when it comes to scaling an experimentation program internally. That’s because there are more opportunities to maximize the customer experience.

I talked to several champions of product experimentation about their programs:

  • Hila Qu, Vice President of Growth at Acorns;
  • Meg Watson, Product Manager at Stitch Fix;
  • Martijn Scheijbeler, Vice President of Marketing at RVShare;
  • And Giannis Psaroudakis, Director, Product at Optimizely.

And the resulting post, “Experimentation in product development: How to maximize the customer experience” provides real-world success stories and examples of potential opportunities that can inspire you to get started.

3. Techvibes’ Spotlight article “Data & Design: How WiderFunnel Cracked the Code”

WiderFunnel Office Design Techvibes Matt odynski
Techvibes’ Matt Odynski came by in January to do a tour and photo shoot of our new office design.

In January, Matt Odynski of TechVibes stopped by our office for a photo shoot and tour of our new space in downtown Vancouver.

The resulting Spotlight article, “Data & Design: How WiderFunnel Cracked the Code,” written by Max Greenwood, provided an insightful overview of WiderFunnel’s services, culture, and people.

WiderFunnel Office Design Techvibes Matt Odynski
A shot of the Burrard Inlet from our downtown Vancouver office windows. Credit: Matt Odynski

WiderFunnel works with some of the world’s biggest brands—such as Mark’s, SportChek, IBM, Square, and H&R Block—to help them gain unique insights into their customers. WiderFunnel can then validate those insights through experimentation in the real world, perfectly mixing both the creative side of a typical advertising agency while also embracing the data and scientific elements needed to deeply innovate the industry.

Max Greenwood

Staff Writer at Techvibes

WiderFunnel Office Design Techvibes Matt Odynski
A shot of the Townhall area, a place for team gathering. Credit: Matt Odynski

3. The 5 pillars of digital transformation strategy at Mark’s: An interview with changemaker, Johnny Russo

Mark’s (formerly Mark’s Workwear House) is a retailer under the Canadian Tire umbrella that has risen to the top in recent years. Not only do their marketing teams receive nomination after nomination for their e-commerce experience, but the company is also leading the charge when it comes to digital transformation.

WiderFunnel Digital Transformation Strategy Mark's
Mark’s is undergoing a digital transformation, empowering every level of business to make data-driven decisions.

In this interview with Johnny Russo, Associate Vice President of Digital Marketing and E-commerce, we get an in-depth look at the five pillars of their digital transformation strategy that has fuelled their growth over the past few years: People, Partners, Culture, Education, and Change Management.

Johnny Russo is a true changemaker. He is passionate about digital transformation strategy and he is a definite thought leader in the e-commerce and retail space.

Having partnered with Mark’s for numerous years, Johnny also outlines how experimentation has underpinned their whole strategy. Testing is now how the team at Mark’s make decisions.

As marketers, we think we know it all. But testing actually confirms that we don’t. If we have an A/B test, one of them might be wrong. And we’re actually telling our senior executives, by doing this, we don’t know which one will win. We think we know, but at the end of the day, it’s based on data and customer experience and what customers want. It supersedes what our thought process is.

Johnny Russo

AVP of Digital Marketing and E-commerce at Mark’s

Read the full post, “The 5 pillars of digital transformation strategy at Mark’s: An interview with changemaker, Johnny Russo.”

4. Winning Optimizely’s “Innovation Partner of the Year” award at Opticon

While in Las Vegas for Optimizely’s annual experimentation conference, Opticon, we were awarded their prestigious Innovation Partner of the Year award and we were ecstatic.

WiderFunnel Experimentation Articles Optimizely's Innovation Partner of the Year
The WiderFunnel team was jazzed about winning the Innovation Partner of the Year for our thought leadership and technical talent.

We were especially honored to be chosen by our colleagues at Optimizely for this award because of our early adoption of FullStack, their server-side experimentation platform.

They also praised our thought leadership in the world of experimentation, including our content, case studies, best practices, and of course, our “State of Experimentation Maturity 2018” original research report, that we co-created.

5. The “State of Experimentation Maturity 2018” original research report

With Optimizely, we surveyed marketers, product managers, and growth strategists at some of North America’s leading brands like Nike, United Airlines, Showtime, American Express,, MailChimp and many more to create the “State of Experimentation Maturity 2018″ original research report.

We also interviewed dozens of Optimization Champions throughout our research to understand their opportunities, their pain points, and their efforts to scale their programs cross-organizationally.

WiderFunnel Experimentation Maturity Research Report
Our State of Experimentation Maturity 2018 was one of our top highlights of the year. That’s because it was all about the work you are doing!

We wanted to know what the most successful organizations were doing right, how they were scaling their experimentation program, and how they were integrating experimentation into their overarching business strategy.

We highlighted the most illuminating findings in our post, “6 Key Insights from the “State of Experimentation Maturity 2018” original research report. And we even got interest from some press:

If your marketing experimentation program consists of A/B testing email subject lines or landing pages, you have a long way to go. But here’s some good news: you aren’t the only one just getting started.

Natasha Wahid, Marketing Lead at WiderFunnel, who spearheaded the research project, also conducted a GrowthHackers AMA in June, diving into the research process and data analysis.

WiderFunnel GrowthHackers AMA with Natasha Wahid
Natasha Wahid, Marketing Lead, hosted a GrowthHackers AMA, which was a must-read in their weekly round-up email.

And our conversations and insights fuelled our content for months afterward. Posts like…

6. Evangelizing experimentation: A strategy for scaling your organization’s test and learn culture

After our research report, one topic that invigorated our marketing team was how to evangelize experimentation at an organization.

After talking to numerous Optimization Champions, we realized that most companies wanted to strategically communicate the value of their experimentation programs internally to incite organizational buy-in from other departments and the Executive team.

It was an interesting take on building a culture of experimentation, a topic that has quickly accelerated over the year.

For this post, we gained insights from leaders in the field including:

  • Alex Birkett, then Growth Marketing Manager and recently promoted to Senior Growth Marketing Manager, User Acquisition at Hubspot
  • Andrew Capland, then Director of Growth and now Director of Marketing at Wistia;
  • Ralph Chochlac, Former Director of Product Management at Student Brands;
  • and Lauren Schuman, Director of Growth at MailChimp.

This comprehensive guide to strategic communications for your experimentation program provides best practices and tactics for scaling your test-and-learn culture.

Read the full post, “Evangelizing experimentation: A strategy for scaling your organization’s test and learn culture.”

7. A tactical guide to creating emotional connections with your customers

We talked a lot about emotional marketing in the last couple of years, but we still saw ambiguity in what creates an emotional connection with customers, especially within a digital experience.

So, we set out to break it down for our readers. We took a real-world example (the WealthSimple website) and analyzed everything from color choices to messaging, from storytelling to animations.

And we also talked to emotional marketing experts, like:

  • Roger Dooley, author and podcast host of Brainfluence;
  • Nick Kolenda, expert and author of numerous specialized guides in psychology and marketing;
  • And Nathalie Nahai, web psychologist, speaker and author of Webs of Influence: The Secret Strategies That Make Us Click.

And it’s one of our most read and shared posts of the year.

Read “A tactical guide to creating emotional connections with your customers.”

8. BCBusiness’ “Office Space: WiderFunnel Embraces Team Spirit”

In July, the WiderFunnel were excited to get the print and web editions of BCBusiness, featuring our new office space. We moved in August 2017 and this year we’ve been featured in numerous publications, including the Vancouver Sun.

Editor Felicity Stone came to do a tour of our space with intern Aleena Deandra and the resulting article showcased exactly what makes the WiderFunnel office so special.

WiderFunnel BC Business Culture
Office Space: WiderFunnel embraces team spirit, via BC Business.

The Vancouver-based company leased a space with spectacular views of downtown and Burrard Inlet, then surveyed staff about what they needed to be productive and creative. The result is a workplace that accommodates various working styles, thanks to wireless technology, so team members can change locations throughout the day. Since employees moved in last August, their happiness scores have increased by 51 percent.

Aleena Deandra and Felicity Stone


Especially interesting to note is that we recently expanded our office by an additional 1600 square feet this past fall!

9. Reaching #21 on Business in Vancouver’s Fastest Growing Companies in BC

At WiderFunnel, 2018 was a year of growth. We nearly doubled our employee count. We expanded our office space. And we worked with more and more Optimization Champions to scale their experimentations programs across their entire organizations.

And it’s probably a big reason why we reached #21 on Business in Vancouver’s list of the Top 100 Fastest Growing Companies in British Columbia.

WiderFunnel Business in Vancouver Fastest Growing Companies in BC
Last year, we were #43 on Business in Vancouver’s list of the 100 Fastest Growing Companies in BC. This year, we were #21!

For context, we were delighted by our #43 standing in 2017, so when we saw our 2018 results we were astonished.

But we’re not done yet. Stay tuned in 2019 for more exciting developments in the world of experimentation.

Get ready for 2019.

The world of business is changing quickly.

Instead of backtracking on failed strategies and tactics, organizations can pave their way to success, proving they are on the right path through experimentation.

We touched upon the upcoming advancements in “The future of digital customer experience: 6 Experimentation trends for disruptive businesses in 2019” and “6 Marketing trends set to take off in 2019.”

As business leaders, we are the cusp of even more transformation in 2019. Because those organizations, that are able to gather insights and spread those insights throughout their organization will change at a pace far greater than they have in the past.

Get ready.

What do business leaders need to consider as they plan for 2019? We’d love to hear your perspective!


Lindsay Kwan

Marketing Communications Specialist

Benchmark your experimentation maturity with our new 7-minute maturity assessment and get proven strategies to develop an insight-driving growth machine.

Get started

Source link

Building the essential marketing technology stack to fuel your experimentation program

Building the essential marketing technology stack to fuel your experimentation program

An experimenter’s guide to marketing technology

Most business leaders today are familiar with Scott Brinker’s famous Marketing Technology Landscape supergraphic:

Widerfunnel marketing tech stack supergraphic
Marketing Technology Landscape Supergraphic (2018)

In 2018, the graphic included 6,829 marketing technology solutions—almost double the number of solutions depicted in 2016. There are literally thousands of technology platforms at your fingertips. And each promises to solve every one of your business pain points.

But technology without purpose and strategy quickly becomes shelfware. A strategic approach to both sourcing and leveraging your marketing tools is essential.

Today’s post details the technology roadmap recommended by Widerfunnel’s Senior Experimentation Strategists. This roadmap highlights the “why” behind implementing certain technologies at certain moments in your organization’s experimentation journey. As well as how to get the most out of these tools by taking a strategic approach.

Note: We will not cover every type of marketing technology, but will focus on the tech stack that should be leveraged by a team that is conducting online experiments.

Strengthening the Technology pillar in your Experimentation Operating System

But first, let’s zoom out.

Your experimentation technology stack is just one piece of your Experimentation Operating System™ (EOS). There are four other pillars, which include: Process, Accountability, Culture, and Expertise.

You must view your tools within the context of the entire system:

  • How do they support you in documenting, unifying, disseminating, and automating your experimentation processes?
  • How do they support you to hold your experimentation program accountable—to properly track and report on program success?
  • How do they support you in collaborating with the organization at-large? In promoting visibility and transparency around experimentation to create a culture of experimentation?
  • How do they support your team of experts in terms of usability, productivity, training, and customer support?
  • How do they integrate and work with your existing technology stack?

As your experimentation program matures, you will likely experience constraints across the different pillars. Technology will not always be your most important constraint. In fact, we speak to many organizations with extremely sophisticated marketing technology stacks that are floundering in areas like process and culture.

Widerfunnel experimentation maturity pillars
WiderFunnel’s pillars of experimentation maturity.

Your experimentation program will only reach its growth-driving potential if it is strong across all pillars.

But today—we are talking technology. If this is your organization’s primary focus or suspected weak spot, you are reading the right blog post.

Technology in the Initiating Phase: Analytics & an experimentation platform

Organizations in the Initiating Phase of experimentation maturity are just getting started. In this stage, an Experimentation Champion is likely working to implement the appropriate technologies and get initial wins to prove the value of an experimentation program.

If your organization is just getting started with experimentation, your primary focuses should be:

  1. Ensuring you have a solid analytics foundation, and
  2. Identifying and selecting the experimentation platform for your needs

Analytics and quantitative research

Clean data for quantitative research is an essential starting point for any experimentation program.

Analytics data will inform the initial story of your customer journey; it will enable you to prioritize test areas and gain insight into your customers’ goals and intentions. We group these into two buckets: Primary analytics and supplementary analytics providers.

Primary Analytics: There are really two key players that provide primary analytics reporting for most organizations: Google Analytics and Adobe Analytics. Together, these two have a lock on around 75% of the market. Google dominates for websites of all sizes, however Adobe can’t be discounted, particularly as a solution amongst high-traffic Enterprises.

Another company to keep on your radar is Heap Analytics. Heap has emerged as a powerful solution known for its ease of implementation and ability to auto-tag all site activity. For companies with limited IT resources, this single-line of javascript implementation is an attractive option.

Supplementary Analytics: While Google and Adobe dominate as primary analytics solutions, there are many emerging technologies you can use to supplement the data your primary tool is collecting. These tools provide advanced solutions for data visualization, marketer-friendly tagging, and prescriptive analytics.

These solutions may not be worth the expense for smaller organizations in early stages of experimentation maturity. But when you start asking very complex questions and dealing with larger data-sets, your primary data analytics tool can become restrictive.

Companies that are looking to take their analytics game to the next level should look into:

With a solid analytics set-up, your team can start to uncover the low-hanging fruit within your digital experience and identify opportunities for testing. You will quickly find, however, that you need to implement an experimentation tool to increase the confidence you have in the changes you are making.

Experimentation Tool

The next piece of the puzzle is a platform to facilitate experimentation. These tools allow you to measure the statistically valid effects of any change you make by limiting the impact of time and audience variables.

When you rely on analytics alone, you are relying on a less than accurate “before and after” measurement system. Making key business decisions based on this type of system can often lead you down the wrong path.

An experimentation tool allows you to run experiences simultaneously to randomized samples of your audience, giving you data you can trust to make decisions.

Before choosing a tool, it is important to identify your program’s requirements and parameters. For instance:

  • What sort of targeting requirements does your website have?
  • Are you going to require server-side testing?
  • Does your development environment allow for third party Javascript?
  • Are you testing on the web or do you need to be able to test in a native application?
  • Do you need Single Page App (SPA) support?

The answers to these questions in combination with your budget should help you select the right tool for your needs.

In the North American market, there are four major players in the experimentation space and many niche options. Each has unique strengths and features to offer depending on the needs of your company.


Optimizely is the market leader by market-share and is considered a thought leader in the experimentation space. They lead the way with their Stats Engine, broad integrations, and expansive product suite; they are considered a one-stop solution for organizations in any phase of experimentation maturity.

Optimizely’s experimentation-first perspective means they are usually first to market with the most innovative feature developments and advancements in the industry, keeping their customer base ahead of the competition. If you are an Enterprise organization, this is the first tool to evaluate. These features do come with a higher price tag though, and Optimizely may be out of range for smaller organizations that are just getting started.

Adobe Target

Also servicing enterprise companies, Adobe Target provides a testing solution to those that are heavily leveraging the Adobe technology stack. Like Optimizely, Target offers solutions to most of the key requirements any company may have. Those with Adobe Analytics and other Adobe products may find additional value due to Adobe’s cross-product integration. However, Target isn’t the most user friendly tool. And if you aren’t a trained data analyst or statistician, its statistical model can leave room for error. If you don’t already leverage the Adobe suite, you will likely not get the full value of Adobe Target.


VWO is a great solution for smaller or mid-market companies. This tool has a much more friendly price-point and allows for all basic client-side testing functionality. They have a strong SmartStats Engine, and built-in variation heatmaps, which are a major bonus. VWO is a great option for organizations with a standard client-side website environment that are looking to prove the value of experimentation. However, companies with very specific technical needs may want to pass on VWO due to their lack of server-side or SPA technology.

Google Optimize

New to the game (sort-of), technology giant Google has also developed an experimentation tool. The obvious advantage of Google Optimize is its native Google Analytics (GA) integration, and its free entry-level price-point. That said, the free tool is rudimentary and lacking in most advanced features. It also limits users in the number of experiments and metrics that can be in the tool at once. Optimize can be a worthwhile solution for GA organizations that are just getting started with basic testing.

Google’s paid version, Optimize 360, unlocks additional experiments and audience criteria that allow for more advanced experimentation. But it is only as valuable as your GA integration. If you have a team of GA experts, then you can do some creative analysis. However, Optimize 360 is likely not worth it for more entry level programs.

Keep in mind, Google Optimize is a sleeping giant. If Google decides to pour resources into the development of this tool, it could quickly become the dominant player in the space.

Sentient AI

If your website is well-suited for multivariate testing (MVT)*, then Sentient AI offers a very interesting solution. Their tool uses Learning Evolutionary Artificial Intelligence (LEAF).

Loosely following the Theory of Evolution, Sentient lets you set thousands of variables and then watch as the tool “breeds” new combinations of variables and sends others to “extinction”. Eventually, it works its way towards the optimal combination. Due to its complexity, this solution is not for everyone. But in the right hands and on the right site you can cover a lot of ground quickly.

*Sites suited for MVT often have many fixed, modular components that are independent of one another.

Client-side versus server-side experimentation

One of the most important questions to consider at this stage is whether or not you need a client-side or server-side solution. Both have benefits.

Client-side execution means the code for your experiment variation will be injected through the browser. While this can be worse for performance, it enables the use of simple WYSIWYG interfaces that allow marketers to make changes without involving developers. One of the other major drawbacks is that you can only test on features that already exist in the DOM.

Client Side Experimentation Infographic
Here’s a diagram showing how client-side experimentation works.

Server-side execution means that the variation code will actually run on your server. The primary benefits of server-side are improved performance, security, and the ability to test features that do not exist in the control environment. Teams can leverage this to roll out new pages, entirely new functionality and prototypes, or make changes on pages with complex server calls.

WiderFunnel server-side experimentation
This diagram showcases server-side experimentation.

Server-side also greatly reduces the time needed to hardcode experiments because the code is already built in your native environment and follows all of your conventions. Server-side experimentation does require more initial investment to install the appropriate SDK but has efficiency and security benefits down the line.

If you are evaluating server and client-side technologies, consider the expertise of your engineers. Because server-side testing is done in your native environment, your team may find it more accessible. Client-side, on the other hand, relies heavily on Javascript and jQuery.

Technology in the Building Phase: A foundation for collaboration

Organizations in the Building Phase of experimentation maturity are bought in on the value of experimentation. In this stage, an Experimentation Champion or team is likely establishing process and building the infrastructure to scale the program.

An analytics tool and an experimentation tool make up the technology stack of a basic Experimentation Operating System. A single team can leverage both to plan, run, and analyze experiments. However, as you work to scale your experimentation program, collaboration becomes crucial. How can you enable experimentation across many teams and business units?

While an individual can test with little documentation, your organization will need solid project management and record keeping to scale up an effective program. In our experience, you will need a specific experiment collaboration system in place to move upward on the experimentation maturity scale.

There are a few things to consider when selecting collaboration tools:

  • How will people throughout the organization submit test ideas?
  • Where will experiment wireframe and design files be hosted?
  • Where will people communicate about an experiment?
  • How will the current status of each experiment be displayed?
  • Where and how will the results be stored?

You may not need to source technologies that address all of these questions at the outset, but you should consider tool(s) that will be able to grow alongside your program. Here are a few popular solutions we have seen:

JIRA: JIRA can really do it all. Although its interface can be confusing at times, JIRA is extremely flexible, allowing you to build a custom process that works for most situations. JIRA is a strong solution if your experimentation program sits within Engineering or Product, since it is likely JIRA is already the tool of choice of your engineers. If your program sits within less technical teams, such as Marketing, you will likely want to turn to something a little bit more user friendly.

Asana: Asana is a strong task management solution, and is the tool of choice for project management at WiderFunnel. It allows users to build consistent templates, facilitates task assignment and communication, and has useful scheduling features. The accessible interface makes this a tool that everyone in the organization can use with ease.

Trello: Kanban boards! Certain organizations love working in a kanban view, and Trello is the leader in this space. For companies that want a visual representation of their experimentation projects, Trello can be a great solution. Plus, it integrates well with JIRA as both come from the same parent company—Atlassian.

Optimizely Program Management: One of the key value propositions offered by Optimizely is the integration with their Program Management solution. Optimizely Program Management allows users to store experiment ideas, vote on priority, track insights, report on program success, and more. Although pricey, Optimizely Program Management is a glimpse into the future of experimentation and insight management.

Traditional Spreadsheets: For teams in the early days of the Building Phase looking to get organized, spreadsheets are definitely a viable option. While spreadsheets often fall short on image storage and communication, they are a free, easy solution for smaller teams tracking a simple program. Keep in mind that scale will likely be a problem here.

As you add more tools to your marketing technology stack to enable experimentation, you’ll want to make sure they integrate nicely. If you’re an Adobe user, your analytics and testing tool will already be integrated—you should ensure that anything else you layer on plays nicely with this suite.

As an alternative to suite solutions, several leaders in the experimentation tech space have formed the Digital Experience Stack (DXS). If you use Optimizely or another tool within this stack, you will want to evaluate other DXS solutions.

WiderFunnel The Digital Experience Stack DXS
The best-in-class tools are partnering to create The Digital Experience Stack (DXS). Source: Optimizely.

Achieving experimentation maturity with qualitative research

Organizations in the Collaborating Phase of experimentation maturity are expanding the experimentation program and collaborating across teams. Finalizing a communications plan and overall protocol for the program is a priority here.

For organizations in the Scaling Phase of maturity, experimentation is a core strategy. Standards are in place and success metrics are aligned with overall business goals, enabling testing at scale.

Game-changing experiment ideas come from customer research. Quantitative input from analytics will help you identify potential pain points within your digital experience, but it only tells a portion of the story. Sophisticated experimentation programs also work to layer in qualitative research—an equally important counterpart that will help you fill in gaps in your understanding of your customers.

The goal with qualitative research is to uncover the “why” behind the “what” that you have observed with your quantitative tool.

Ideally, your experimentation team will have all of the following qualitative tools at their disposal:

  • Scrollmaps
  • Clickmaps
  • Movemaps
  • User polls
  • Surveys
  • User session recording
  • Interviews

User Engagement Tools

Many organizations start with user engagement tools. These tools—including scrollmap, clickmap, heatmap, and user session recording features—help you visualize the visitor experience. They are useful because they are passive (requiring no additional action from your visitor) and are often low cost and easy to use.

When analyzing data from these tools, you’ll want to consider:

  • What % of your visitors is seeing specific important content?
  • What % of your visitors is scrolling past the fold?
  • Is there a particularly steep drop off after seeing a specific piece of content?
  • When comparing multiple calls-to-action, which are more commonly clicked?
  • What valuable information might your visitors be missing?
  • Where should you run your next test? (You may not want to test an element that few users are seeing)

Polls & Surveys

Polls and surveys take user research a step further by actually addressing specific questions to your audience. They give you the opportunity to ask questions while your user is in the shopping mindset and is evaluating your product. These require some action from users, but can often uncover much deeper insights (such as pain points within the customer journey you may have overlooked).

Exercise discretion with polls and surveys. Although they can provide rich customer feedback, they can also distract from your primary conversion goal.

Recommended tools to evaluate include:

Advanced experimentation & personalization: Data and customer management technologies

We hear shouts of “hyper-personalization” constantly—a one-to-one customer experience is the pinnacle for many organizations today. Of course, this relies on the existence of underlying data to define what makes an experience “personal”. Most organizations do not have the proper technology in place to enable this.

If your business has been struggling to do effective personalization, you should 1) interrogate your overall personalization strategy, and 2) look closely at how you are managing your customer data.

The proper use of a robust customer data platform (CDP) is a key component that differentiates mature experimentation programs from the immature. These platforms open up the world of audience management, boosting your ability to identify and target high value audience segments and plan test strategies around these groups.

Customer Data Platforms (CDP)

The CDP Institute defines a Customer Data Platform as “packaged software that creates a persistent, unified customer database that is accessible to other systems.”

A major benefit of a CDP is the ability to deliver a more effective customer experience and more impactful marketing messaging. (Which is really the goal with personalization). Your customers want a consistent experience across all of the channels and devices they’re using. They don’t want to see an ad for something they have already purchased. A CDP allows you to gain a complete view of your customer and deploy a consistent experience across touchpoints.

It is important to note, however, that a CDP is only as useful as it is comprehensive and actionable. This means that the number of data sources available to your CDP, as well as the number of execution integrations are both critical.

If you aren’t collecting data from multiple sources—website, mobile app, customer service system, in-store behavior, beacons, etc.—you may not be ready for a CDP. As well, if you can’t activate this data across multiple touchpoints—website, in-store, customer service interactions, etc.—you will not unlock the full potential of a CDP to provide a truly unified customer experience.

A CDP is not a personalization tool. However, it provides the data that will allow you to get the most use out of your personalization and experimentation tool.

If yours is a mature program and you are looking to enhance your personalization efforts or improve your overall data management efforts, Tealium’s Universal Data Hub is a great choice. Evergage is also a strong solution, combining a CDP with real-time personalization capabilities. If you already have a data platform in place, Evergage also functions as a powerful standalone personalization platform.

If you are looking for other tactical personalization solutions, you should also evaluate Optimizely Personalization and Dynamic Yield.

The business world has come a long way from the Mad Men era of focus groups and hunches. Today, technology enables experimentation at scale. It allows organizations to test one experience against another, achieving a statistically confident result. Which greatly reduces risk in decision-making.

However, it is vital that you define your experimentation strategy and ultimate objectives before sourcing your technology stack. Choose tools that fit these objectives, as well as your organization’s culture and the skill levels on your teams.

A final note: Before sourcing more tools, make sure you have a clear idea of what your program constraints really are. You can install every tool on the market, but that doesn’t mean you will have an effective experimentation program. First, you need a map of how you plan to get your program from A to B; technology is simply what you will use to pave the roads along the way.

What tools make up your marketing stack for experimentation? How do these tools support your overall experimentation efforts? Are your favorite tools on this list or did we miss them? Leave your thoughts in the comments section below!


Michael St Laurent

Director of Experimentation Strategy & Product Development Lead


Natasha Wahid

Marketing Manager

Benchmark your experimentation maturity with our new 7-minute maturity assessment and get proven strategies to develop an insight-driving growth machine.

Get started

Source link

How The Motley Fool leverages experimentation as a growth strategy: A case study

How The Motley Fool leverages experimentation as a growth strategy: A case study

Case Study Executive Summary

Company: The Motley Fool
Industry: Media; Financial Services
Business Model: Premium Service Subscription
Experimentation Champion: Nate Wallingsford, Head of U.S. Marketing Operations and Optimization at The Motley Fool

Business Need:

  • Increase paid user acquisition to hit revenue targets
  • Increase user engagement with premium services
  • Increase member retention
  • Increase customer lifetime value


  • Process-oriented experimentation featuring proper Design of Experiments
  • Collaborative experiment ideation between The Motley Fool and WiderFunnel
  • Internal socialization of experimentation at The Motley Fool
  • Staff augmentation (UX / UI Design and Web Development)


  • Millions of dollars in uncovered additional annual revenue
  • Key customer insights generated and leveraged throughout The Motley Fool’s funnel
  • Unified experimentation efforts across The Motley Fool’s marketing and product teams

Controlled experiments can transform decision making into a scientific, evidence-driven process—rather than an intuitive reaction. Without them, many breakthroughs might never happen, and many bad ideas would be implemented, only to fail, wasting resources.

The full story: How The Motley Fool uses experimentation to drive growth

When Nate Wallingsford stepped into the role of Director of Conversions, Acquisition Marketing at The Motley Fool 3 years ago, he knew that experimentation would be a primary focus.

Nate understood that the ability to provide data-backed recommendations for digital experience improvements is crucial. And for a multimedia company like The Motley Fool, whose website receives millions of unique visitors each month, even a slight increase in customer acquisition rate could have a massive impact on the bottom-line.

Today, Nate is the Head of US Marketing Operations and Optimization at The Motley Fool, and the company is testing across the entire customer journey and multiple teams.

While experimentation has always been celebrated at The Motley Fool, Nate’s work in partnership with WiderFunnel has led to increased visibility for the experimentation program, massive revenue gains, and—perhaps most importantly—actionable customer insights.

This case study explores how Nate and his team are using experimentation as a fool-proof (pun intended) growth strategy.

The business context

As mentioned, The Motley Fool is an established multimedia financial services company.

Their website consists of pages on pages of useful, free content for investors; the company generates revenue through various stock, investing, and personal finance premium services. The Motley Fool’s funnel consists of three primary areas:

  1. User acquisition or front-end marketing: The goal being to encourage non-paying members to sign up and become paying members
  2. Upsell or back-end marketing: The goal being to encourage members paying for lower-tier services to purchase more premium services
  3. Product: The goal is to increase member engagement with the service(s) they’re paying for and increase member retention

Each area of the funnel is the responsibility of a different team within The Motley Fool, but everyone is focused on creating the best customer experience and increasing customer lifetime value.

Across the teams, we’re all really focused on customer lifetime value (LTV) for every stage. Whether it’s someone’s first product with us—getting them into a higher tier. And on the product side, the more we get people to renew, the higher their lifetime value. It’s a win-win for both the customer and us—they get great stock advice and recommendations and we get to retain them as a customer.

Nate Wallingsford, Head of US Marketing Operations & Optimization at The Motley Fool

As the leader of the user acquisition team, Nate’s main focus was to hit his revenue targets by increasing paid subscriptions. And experimentation was his tool of choice.

Finding the right experimentation partner

From the outset, Nate had almost everything he needed to build a successful experimentation program within his team: senior-level support, high website traffic, plenty of ideas for improvement. But he was lacking a process.

Nate was running what he refers to as “good-idea tests” – you have what you think is a good idea, you test it, it wins, loses or is inconclusive. If it wins, hooray! You can implement changes. But if it loses, it’s back to the drawing board.

As a self-proclaimed process-oriented person, he knew there had to be a better method. It was in this moment—searching for experimentation processes on Google—that Nate stumbled upon WiderFunnel Founder Chris Goward’s book, You Should Test That! He bought it, read it, and loved it.

I thought, ‘This book is awesome.’ I’m going to start building out our optimization program for acquisition around the WiderFunnel methodologies.

— Nate Wallingsford

And he wasn’t kidding. Nate took everything he’d read in the book, as well as resources from the WiderFunnel website: The LIFT Model®, what makes a great hypothesis, the Infinity Optimization Process™, the PIE prioritization framework – and built a Trello board of the WiderFunnel process as he interpreted it.

Growth Case Study Example WiderFunnel LIFT Model
Nate built The Motley Fool’s experimentation strategy using several WiderFunnel frameworks, including the LIFT Model.

Nate understood experimentation from day one. It was fantastic. He was already familiar with the LIFT Model and experiment design principles. We were able to jump into collaborative experiment ideation and strategy right away. You know, the fun stuff.

James Flory, Experimentation Strategist, WiderFunnel

Ultimately, Nate decided to partner with WiderFunnel to ensure his program would be as successful as possible: To solidify his use of these processes internally. To gain access to additional design and web development resources. And to collaborate with expert experimentation strategists who are testing across industries every day.

The power of fresh perspective

With every new client, there is a discovery phase. In the early days of our partnership, Nate and the WiderFunnel team dug into The Motley Fool’s most important user acquisition goals and conducted initial analyses of

One of the first things we noticed was the lack of a prominent call-to-action to sign up for The Motley Fool’s premium services. Users were landing on the site and engaging with content, but there was no clear path to conversion. In fact, there was almost no indication that there were professional services available.

Nate and his team have a ton of knowledge about their product, their marketing, and their customer, but they had overlooked a seemingly common-sense improvement. This happens to marketers constantly. We are so close to our day-to-day, so wrapped up in our way of thinking, that we miss simple solutions.

Which is why a partner with a fresh perspective can be essential.

The first experiment WiderFunnel ran with us was to add a call-to-action button to our desktop site-wide navigation. It was super simple and seemed like a no-brainer, but the impact was insane. After that, I knew this was a good decision. And I knew it was going to be really cool to work with [WiderFunnel].

— Nate Wallingsford

The experiment details

For this experiment, we added a “Latest Stock Picks” call-to-action in the navigation. This CTA replaced a dropdown menu labelled “Stock Picks”. The assumption was that Motley Fool users are looking for stock-picking advice, specifically.

WiderFunnel Motley Fool Experiment Example
The Motley Fool’s original site-wide navigation—aka “The Control”.

Hypothesis: Creating a clear “Latest Stock Picks” CTA in the site-wide navigation will cause more users to enter a revenue-driving funnel from all areas of

The variations

Two variations were tested. Each featured the “Latest Stock Picks” call-to-action. But in each variation, this CTA took the user to a different page. Our ultimate goal was to find out:

  1. If users were even aware that there were premium paid services offered, and
  2. Which funnel is best to help users make a decision and, ultimately, a purchase

In variation A, the “Latest Stock Picks” call-to-action sent users to the homepage and anchored them in the premium services section. This section provides detail about The Motley Fool’s different offerings, along with a “Sign Up Today” call-to-action.

Widerfunnel Motley Fool Experiment Example
Variation A featured a new call-to-action button, which anchored visitors lower on the homepage.

With variation B, we wanted to experiment with limiting choice. Rather than showing users several product options, the “Latest Stock Picks” call-to-action sent them directly to the Stock Advisor service sign up page; this was Motley Fool’s most popular service.

Widerfunnel Motley Fool Experiment Example
In variation B, the CTA took users to a specific product page.

Both variations beat the control. Variation A resulted in an 11.2% lift in transactions with 99% confidence and variation B resulted in a 7.9% increase in transactions with 97% confidence.

Interestingly, because variation B was built on variation A using factorial design, we were able to see that this change actually decreased transactions by 3.3%.

What is Factorial Experiment Design?

Factorial Design is a method of Design of Experiments. Similar to multivariate (MVT) testing, factorial design allows you to test more than one element change within the same variation. The greatest difference is that factorial design doesn’t force you to test every possible combination of changes.

Instead of creating a variation for every combination of changed elements, you can design an experiment to focus on specific isolations that you hypothesize will have the biggest impact or drives insights.

Learn more, here.

What were the insights?

In The Motley Fool’s initial experience, users may have been unsure of how to sign up (or that they could sign up) due to lack of call-to-action prominence on the original site-wide navigation. Users also seemed to prefer some degree of choice over being sent to one product (as seen with the decrease in transactions caused by variation B).

Following the customer insights

One of the biggest problems with the “good idea” testing that Nate had been doing is that it prioritizes conversion rate lift over insights, over learning. If an experiment won, great. If not, it had zero value. All that effort to ideate, design, and run the test was wasted.

But a great experimentation program will generate insights with every single test. And that’s what Nate began to build with WiderFunnel.

One series of experiments was particularly fascinating. It generated a core customer insight that The Motley Fool is still leveraging and validating throughout their digital experience. The initial idea was to leverage an extremely common persuasion principle: social proof. Adding social proof to any customer experience is widely accepted as a “CRO best practice”.

The experiment details

A secondary top-of-funnel metric for The Motley Fool is email sign-ups for the company’s email marketing list. For the first experiment on this funnel, we focused on the email capture modal. In one of our variations, we added a social proof statement: “Join over 121,837,512 other Fools who have come to The Motley Fool for investing insights.”

This tactic has worked for other WiderFunnel clients in the past, encouraging more users to enter a revenue-driving funnel. In this case, however, the variation tanked. It resulted in a -11.2% decrease in sign-ups.

Widerfunnel Motley Fool Experiment Example
In this experiment, social proof tanked.

In this experiment, social proof resulted in an extreme reaction from users, indicating high sensitivity around this persuasion technique. One theory was as follows: Rather than being comforted by the fact that others trust The Motley Fool, prospective customers may actually be looking for exclusivity.

Nate and WiderFunnel Strategist, James Flory, wanted to understand further. They ran another experiment, this time on the primary landing page for the email capture funnel. But they leveraged a different form of social proof: a customer testimonial.

Widerfunnel Motley Fool Experiment Example
The original email capture landing page.

The Hypothesis: Adding testimonials will make users trust this page as a place to submit their emails and improve email capture rates.

Widerfunnel Motley Fool Experiment Example
In the variation, we added social proof in the form of a testimonial.

Again, Motley Fool users responded negatively. The social proof variation resulted in a slight decrease in conversions. Seeing this result, Nate thought about the testimonials splashed across the customer journey and wondered: Should we remove social proof throughout the funnel?

“We had started to test injecting social proof into our lead capture pages. And we saw a drag on conversions any time we did that. We tried adding social proof and trust elements on our video sales letter pages. That had a drag on conversion. And we thought it was strange. Adding social proof seems like a best practice, an industry standard,” explains Nate.

“So, we thought, let’s experiment with this on our order pages. These are at the bottom of the funnel, right before purchase. What happens if we remove the testimonials and social proof we have on order pages? Will we then see a lift because earlier in the funnel we saw a drag across the board?”

We ran another experiment, this time on the order page—the stage of the funnel that includes the point of purchase. In the variation, all customer testimonials were removed. This variation performed terribly, decreasing transactions, average order value and revenue per session. However, it generated several actionable insights:

  • Previous learnings indicated social proof had a negative correlation with conversion rate. This experiment challenged that insight.
  • It may be that, in the early stages of the user journey, users are not yet in a purchase state of mind and still crave exclusivity.
  • Early stages of the funnel don’t hint at a paid service or subscription, but adding testimonials may put the thought of an upcoming sales pitch into the user’s mind, possibly triggering an exit or increased wariness.
  • Inversely, when a user is exposed to a purchase decision, they respond positively to social proof which may reduce anxiety and increase trust and confidence in their decision.

That was really interesting to see. Even though we had a decrease in conversion rates across all three experiments, they generated this insight that social proof and testimonials are huge at the point of purchase, but may need to be avoided at the top of the funnel.

— Nate Wallingsford

This series of experiments points to the importance of experimentation in general. If Nate had simply made changes to based on best practices, he might have seen conversion rates drop with no understanding as to why.

And if he hadn’t been leveraging an experimentation process to understand where to retest and revalidate insights (in this case, the threshold and elasticity of social proof), he might’ve just removed social proof lower in the funnel based on the initial experiment results, assuming that social proof doesn’t work.

Socializing experimentation: The importance of gaining visibility

Every marketer and product owner has growth targets they are trying to hit. Which is why achieving positive experiment results is hugely important. But visibility is crucial to the longevity of any experimentation program—on both winning experiments and ‘losing experiments’ that generate learnings.

Nate’s goal has always been to promote a culture of evidence-based decision-making at The Motley Fool.

Early on, Nate realized that the insights gained through process-based experimentation were a firestarter for even better tests. He wanted to spread this knowledge throughout the organization, so he began compiling his experiments and insights into a monthly email newsletter.

At first, Nate was just distributing this newsletter to the U.S. acquisition team. But people began to forward it on, and more Fools became interested in joining his distribution list. So, he began to scale this communication to other teams.

This newsletter became a key resource for other teams at The Motley Fool—specifically teams with lower website traffic. These teams lack the traffic volume to test at the same velocity as the acquisition team, but are able to leverage Nate’s insights and results to implement new experiences on their sites.

Today, Nate and his colleague Lauren conduct a weekly standup on experimentation. Attendees come from across the company—from marketing, technical, and editorial teams. This constant communication generates buzz and momentum around experimentation at The Motley Fool and is a key piece of Nate’s strategy.

The future of experimentation at the Motley Fool

At the beginning of this partnership, Nate was looking to leverage WiderFunnel’s expertise in experimentation and augment his resources to scale The Motley Fool’s experimentation program quickly. The relationship has since morphed into a highly collaborative partnership. Today, Nate and James feed off each other’s insights and ideas to develop new tests and experiences together.

The test ideation, optimization conversations, and overall rapport [between us and WiderFunnel] is exceptional. I feel like I’m having these conversations with my colleagues, not an agency.

— Nate Wallingsford

Recently, WiderFunnel and The Motley Fool expanded their partnership to help drive testing strategy within The Motley Fool’s product experience. This aligns perfectly with Nate’s priorities for experimentation, which are:

  • To enable deeper collaboration between the Marketing and Product teams and unify the new member journey from purchase to product experience
  • And to optimize The Motley Fool’s mobile experience and improve monetization

“Things change quickly at The Motley Fool,” explains Nate. “I always try to prioritize the experiments that have the biggest potential business impact. It’s a big part of what has made our program successful, and will be a continued focus for our team.”


Natasha Wahid

Marketing Manager

Watch this on-demand webinar and discover how Nate and his team are leveraging experimentation to uncover massive revenue gains and actionable customer insights. And learn how Nate has worked to gain visibility and create excitement around testing.

Watch the Webinar >>

Source link