Why I Always Test Multiple Outcomes On A/B Split Tests

Like any good online marketer, I continually split-test as many variables as possible on my websites.

For my largest business, NeverColdCall.com (which sells my ‘Never Cold Call Again’ sales training system), the sales process is typical of that of an information product, beginning with a landing page offering an incentive for people to sign up for my email list, in the form of a free 37-page PDF download of the book portion of the package, followed up by a sales page along with an extensive follow-up email series.

As expected, most sales are made anywhere from a few days to several months after the initial signup, via the follow-up emails. As a result of this I moved most of my focus over the past year, away from optimizing the site itself to working on my follow-up emails.

However, a couple of weeks ago I was in front of the camera shooting a new TV commercial and figured it wouldn’t hurt to take an extra minute and bang out a new homepage video to test, now that I’ve “moved up in the world” a bit and have access to better production quality.

I let the test run for about 3 days. Since the video was on the homepage promoting the free download and email opt-in, that’s what I tested as my primary outcome, and here are the results:

The new video is represented by variable #2, and as you can see, it slightly edged out the original. The results weren’t impressive by any means but they should equate to about an additional 750 opt-ins per month for me, which, based on averages, equates to about 56 sales and around $6,750 in revenue.

Not bad for a “quick” video test.

HOWEVER: Here is the real reason I’m writing this today!

Anytime I run a split test to try to improve opt-in rate, I always add sales conversion as a secondary outcome.

Why?

Because, time too numerous to mention, I’ve had a wildly successful split test on the opt-in side that seriously exploded opt-in rate, but destroyed sales conversion!

In this case, however, the opposite was true. In the above screenshot you see the tiny improvement in opt-in conversion. But look at what happened to my secondary outcome, sales conversion:

As you can see, this time the results were not insignificant at all – by changing the homepage video, sales conversion on those initial visitors nearly doubled!

Why?

As far as I can tell, it’s because the original video pitched visitors to get the “free download,” which is somewhat vague. The second, new video, specifically says to download the “37-page sneak peek.” In other words, I’m making it perfectly clear that what they’re getting is just a teaser, and not the full version of the book.

The original video apparently was attracting either freebie-seekers, or people who mistakenly believed that they were getting the whole nine yards in the form of a free download.

The new video presumably is attracting visitors who know full well that they’re just getting a teaser, are still interested enough to want to see the teaser, and who understand full well that anytime you download a teaser, a sales pitch to buy the full version immediately follows. As a result, they expect to be sold, and are not annoyed when it happens.

Another variable is the fact that the video itself is better quality, the camera is angled so that I am looking down at the camera (a psychological trick that causes the viewer to view you as more powerful than they), and quite frankly because I’m in better shape than in the original video: Dropping 20 pounds and having a thinner face creates the illusion that I’m taller, again another enabler of the power principle.

In summary: The lesson I want you to take away from this is no matter what you test on your website, always always always measure its effect on ultimate sales conversion. The results may surprise you, and you will frequently avoid making a fatal mistake!