How to Use Qualitative Data to Come Up with A/B Testing Ideas

Analytics and split testing tools can only tell you how many visitors convert or leave your site. Figuring out WHY they do what they do is the holy grail of conversion rate optimization. You’ll learn how to use tools like Qualaroo and your Aweber Autoresponder, you can get priceless insights about what visitors think about your site.

Video Transcript

Hey. What’s up, everybody? It’s Brian Dean, from Quick Sprout. In this video, I’m going to show you how you can get ideas for A/B tests using qualitative data. One of the best ways to get qualitative information from site visitors is to use a tool called Qualaroo, which is basically a service that allows you to ask questions directly to site visitors.

The first thing we want to do is head over to and create an account. When you setup your account, they’re going to give you a JavaScript code like this that you need to install on your website in order to use Qualaroo. What you want to do is head over to your WordPress dashboard, hover over Appearance, and click on Editor. Under Templates in the right sidebar, you want to choose Header. Next, look for the closing head tag in your theme source code, right here. Head back to Qualaroo, copy the Qualaroo script, and paste it right after the close head tag. Finally, click on Update File, and you’re all set. Your Qualaroo code is now live on your site, and you can setup your first questionnaire. To do that, head back to Qualaroo again and click on Create New Survey. First, give your survey a descriptive name. For example if you run an e-commerce site and for some reason your tennis racquet category page wasn’t converting well and you wanted to know why using Qualaroo, you could call your survey Tennis Racquet Category.

The next thing you want to do is enter a question text. This is a question that you’re going to be asking visitors. There’s no right and wrong way to ask a question, and there are literally thousands of possible questions, but I just want to go over what not to ask. You don’t want to ask people, “Are my prices too high? Why aren’t you buying?” things directly related to the conversion. You want to ask them things like, “What problems are you having? How do you think this page could be improved? Are you finding everything you need?” Those are the questions that allow you to get qualitative information about your visitors without being intrusive and asking them why they aren’t buying your products and services.

Once you’ve come up with a question, your next step is to determine your answer type. The most common, the least intrusive, and the most likely to get you answers is the checkbox option. For example if you were going to ask them, “What do you wish you saw in this page?” and you can include something like user reviews, price comparisons, things like that, and you can include your answers here. Instead of Answer 1, you’d put Reviews, and they could choose that from the list. If you want to get really qualitative information, you can give them a text-based answer. A lot of times people will surprise you with things that they want to see on a page or something they’re thinking when they’re on the page that you might not think to include as a multiple choice. You might want to test several different types of surveys to see which one gets you the best qualitative information.

Once that’s all setup, click on Next. Next, enter the url of the page that you want the survey to appear on. Again, if you were looking to find out more information about why people weren’t converting on your tennis racquets category page, you’d want to enter the url of that page here. Next, you can determine how much and what segments of your traffic are going to take the survey. In most cases, you want 100% of your visitors to be able to participate in the survey, that way you can get the most data. If for some reason you only want to see why one traffic source isn’t converting or why one traffic source is converting, you can do that by choosing one traffic source. For example, search engines; so you could choose Google, Bing, or other search engines, and even get into specific terms if you have that data available.

Next, it’s time to determine when the survey will be displayed. You usually don’t want to display the survey right after the page loads, because you’re not really going to get any insights from people who have seen the page for the first time because they just landed there. Usually, you want to make sure you wait at least 10 seconds before displaying the survey. Next you can determine how often the survey should be displayed. If you really don’t want to annoy people, you can choose Only Show Once per Visitor. If they see it once and they don’t participate, you cannot show it to them again, or you can show it to them until they actually give a response. Because Qualaroo hangs out in the lower right-hand corner of the screen, it’s not too intrusive, so this option is actually just fine. If you really want to stay on the safe side, you can only show it once per visitor.

Once that’s all setup, click on Next. Next, it’s time to design your survey, or the nudge as Qualaroo calls it. I recommend keeping it pretty much like this. Even though it looks plain, it is what people recognize online, because Qualaroo is used on quite a number of authority sites. It’s likely that some of your users have already seen this, which makes it more likely that they’re going to trust it and actually enter answers in here. Once you’ve adjusted the design so it looks good for your site, click on Save. If everything looks good, click on Activate This Nudge, and the nudge will go live on your website.

The next strategy to get qualitative information for A/B testing ideas is to actually use your auto-responder sequence. What I recommend that you do is the first auto-responder messages that new subscribers get asks them a question. What you want to do is set it up so it looks something like this. Your first auto-responder should thank them for signing up, and then ask them a question, preferably something that they’re struggling with or having trouble with. If you ran a tennis blog or a tennis e-commerce site, you could ask them, “What’s the one product that you wish you had,” or “What issues are you having with your tennis swing?” Then you can incorporate that information and the language that they use into your website’s copy for split testing, which is very, very powerful. Once you speak their language, it’s much, much easier to convert. Like anything, you want to test it, and a great way to get qualitative information for language that you can use in your split tests is using your auto-responder.

That’s all there is to using qualitative information to get ideas for A/B testing. As you can see, Qualaroo is a great tool, but you can also use your auto-responder sequence to find out what your customers are struggling with, and then you can incorporate that copy and those insights into your split testing. Thanks for watching this video. I’ll see you in the next one.