A Refresher Course on Testing
We came across a post from Hubspot that serves as a good refresher course or primer on the basics of testing in digital marketing. The piece covers the differences between A/B and multivariate testing in measuring the performance of Web sites, though the concepts also apply to various forms of online advertising.
So what are the differences between these types of tests and how do you know which is right for you? Technologists and serious analytics pros will debate the precise definitions, but at the most basic level, A/B tests evaluate the differences between entire pages, banner ads or emails. Meanwhile, multivariate testing focuses on elements within individual pages, ads or emails.
As a general approach, companies will often utilize A/B tests in the earlier phases of digital marketing or Web analytics programs to determine the best overall layout or design for their site. Typically, these tests require less traffic for valid results so action can be taken quickly to create the right experience for your audience.
Generally speaking, once they have established the overall layout they want, advanced marketers with more extensive and complex sites will turn to multivariate testing to optimize the many elements within their pages. For instance, they will seek to understand how different individual (and combinations of) offers, headlines, images, copy, color schemes, etc. impact the goals for a specific page. They will use the insights they generate to make small but significant tweaks to their content or campaigns. It’s important to note that multivariate tests usually require a higher volume of page views and user sessions to generate meaningful results.
In the past, conventional wisdom held that testing was expensive and complex. Thus, it was only for big companies with big analytics budgets, huge software platforms, and teams of statisticians. But today, digital marketers can learn a lot for a relatively small investment. In fact, free tools like Google Analytics have pretty strong testing capabilities.
As companies evolve their testing culture and capabilities, they’re able to tap into even more advanced features of their testing platforms. They can use segmentation and personalization to understand what site designs and elements work best for different types of users – current vs. new customers, visitors from different states, visitors who come to the site from social media vs. search, etc. – and automatically deliver the right experiences to different types of visitors. That’s when things get really exciting.
We are great believers in the concept of “test and learn.” In fact, we believe it is a central pillar of effective Web analytics and digital marketing. That’s why we encourage our clients to “always be testing” and to experiment with different types of tests depending on their needs and goals. Testing and learning are relatively easy and cost-effective, and can produce real value. For example, basic A/B comparisons of different home page treatments can give quantifiable evidence about which is more compelling, “sticky” or effective in driving desirable behaviors. The same goes for offers in banner ads on social media sites, alternative formats for email newsletters, or different check-out flows for retail sites.
Thinking about new types of content to offer your customers? Testing is a great way to find out what users may be looking for or would be interested in hearing about from you. Of course, the best tests will come out of a strong process. For example, have your team generate a few new content ideas, shape a hypothesis about which consumers might like it, test it with different executions and measure the results against a few clear metrics. Maybe you’ll hit on a new way to connect with target audiences and meet your business goals – or maybe you’ll find out your current offerings are already attractive to users.
Testing is only growing more important as channels and devices continue to proliferate. Remember testing can be applied almost anywhere – with existing sites and popular search engines, on Facebook and Twitter, even with iPhone apps and mobile campaigns. You could test a small community outreach campaign on social media sites or a few different keywords. The results will give you some basic insight into the impact of different variables, but may also uncover hidden and valuable opportunities.
Because digital marketing is still a new and rapidly evolving discipline, improvement requires that organizations are able to understand what works and doesn’t work, and where the improvement opportunities lie. And that’s what testing and learning is all about.
(Bonus: Check out a bit more about how testing helps analytics and digital marketing pros go back to school and further their Web analytics education here.)
