home> articles > Why Video can help improve Data Mining results
Why Video Testing offers insights which can help web site data mining
By Todd Follansbee, www.webmarketingresources.net
Abstract - adding direct user video testing to data mining can offer complementary and powerful insights which provide clients with the means to better improve conversions and site results.-----
Note: I welcome feedback from the KDNuggets readers! Please send me your thoughts and critiques. I recognize that this may be controversial and my intent is not to annoy but to stimulate discussion and develop the concepts further. Your responses will be kept confidential. Please email me at todd@webmarketingresources.net or call if you prefer.
Until you have seen user video testing, you may be skeptical about the power of what it can teach us about how people behave on web sites. By using the right approaches in video testing for task completion tests, and probing for insights and reactions, you will learn what is and isn't working on your site and why.
We know from independent research that user tests with only 6 people will reveal over 90% of a site's problems.
In addition to a better understanding of site problems, user testing can reveal a range of solutions to these problems from the users themselves. While it is true that mining logs can show which page or page element may work better, direct user testing allows us to ask why they prefer one and, what might work even better. Combining data mining with direct user testing done by an experience User Experience architect, can deliver results and an ROI which engages clients and improves site success and conversions.
Video Testing can also be quick
We are also promoting a new version of video testing which we call Rapid Deployment Testing. It is designed to draw from a local pool of available candidates and run tests within hours of designing the test. This type of testing is designed to explore one or two specific processes. During the design process, information architects and site developers often come up with questions about the latest best practice thinking, in part because these practices continue to change and improve. By testing one or two simple procedures, in a matter of hours we can gain actionable insights. For example: What is the key messaging on a landing page? Is the call to action visible enough? Why are people missing a particular message? How best can I integrate upsells into the purchase process?
As an extension of Rapid Deployment testing, we recommend that clients save and categorize video clips of certain procedures so that stakeholders can review commonly tested behaviors as new questions arise. Having these review materials readily available to designers will improve their design process, without the need to conduct new tests, thereby improving the quality of the prototypes and speeding up design. We have built a corresponding wireframe “best practices” design library which we update regularly. This allows us to rapidly specify and prototype test, design procedures. Building a best practices library reduces design questions, improves prototypes, and shortens design time and, if updated, will allow for the library to be continually improved as each stakeholder learns more and shares that learning.
Smaller Sites Can Benefit Immediately
Metrics testing is only helpful for sites if they have significant enough traffic to conduct a test. Data experts recognize that the longer a test takes, the less reliable it is and, ensuring valid statistical significance can be a problem when multiple variables are considered and traffic is nominal. Sites which lack the traffic to obtain statistical significance should consider video testing to gain understanding of site problems as traffic and sales build.
Video Testing is Impactful
We know that numbers are hard to argue with and that data can be compelling; however, Americans in particular are greatly swayed by video. If we see it on the small screen – we believe. Many times I have watched reluctant clients view several short clips showing user frustration and annoyance and they become convinced to make necessary site changes. In fact, a peer ran a video test where she actually tested the company execs on their own site. The problems they exposed (the execs could barely use their own site), promptly resulted in seven figure funding for a full usability lab.
Video Testing Works as Well on Prototypes as it does on Existing Sites
There is no metrics testing available for prototype sites yet the failure to test early and often has shown that we are doomed to costly design changes on the back end. Prototyping tools allow us to build realistic test “sites” which behave almost exactly like active sites. The agility of these design tools allows testing multiple graphic treatments and conversion paths of even relatively large sites in only hours. Video testing quickly allows us to measure the relative merits of each possibility.
Video Testing Measures Graphic Impact
It is very hard to measure how a site's graphics will impact customers from metrics testing alone. In video testing, we ask about the look and feel of a site by carefully chosen, probing questions which measure overall impressions and brand impact. Metrics testing can compare which graphics may get more clicks, but why one works better is best determined by video testing. Understanding the initial impression of a home page is possibly the most important element in site conversion and it is best done face to face.
Video Testing improves accuracy
I test in part because I know my limitations. I know from over 10 years of experience that people will not consistently react the way I expect. That they will not always understand what I want to communicate. Every time I test, I always find significant benefits and the site gets improved. When I am using metrics testing alone however, understanding why people may click on one thing or another is difficult to assess. This does not discount the facts we learn from data mining, it simply says that we can gain additional insights from direct user testing.
Video Testing is also reliable
Metrics testing is reliable of course however; there remains a significant margin of error in traffic reporting. We cannot predict if future browser changes or changing cookie/privacy considerations will undermine web traffic reports in the near future. With video testing what you see is what you get. The biggest criticism is that the test sample is usually small. I agree, however the research on sample size is very clear. Even small samples turn up most problems (as I said over 90%). It must be emphasized that video testing does not seek to establish statistical significance which is a different thing and why data mining is still essential. The point is that, done in conjunction with advanced data mining techniques, it offers insights into the whys of certain behaviors. Working with testing consultants who have tested and observed hundreds of users certainly adds to the reliability of the interpretations as well.
Video Testing Gives Understanding on an Emotional Level
Buying is an emotional process. We think it is a calculated procedure, but it is much more emotionally based than most believe. Video testing gives an immediate and honest understanding of the emotions of buying. The elements of: risk, incentive, motivation, anxiety, frustration and trust are all things we explore in direct testing and they can be challenging to infer from data mining.
Video Testing Provides Solutions
Metrics testing shows that problems exist, but the way to find solutions with metrics is to test solutions by either A/B testing or using a multi variable optimization tool. Metrics testing can demonstrate that a call to action can be improved. However, video testing will immediately show when it is the message or the persuasiveness of the site that is the problem instead of the actual call to action. Metrics testing can also easily test multiple messages, however video testing might identify the problem as being in the initial message and if the tester listens carefully and probes for understanding, you may be rewarded with a variety of suggestions as well. It has helped in graphics, page elements, design, processes, and of course messaging.
Competitive advantages
When we can't get access to a competitor's site traffic reports or PPC statistics, user testing can allow us to compare for example, how compelling another site may be, what about it works and why they may do a better job of getting conversions.
Video Testing is also a Pay per Click Analysis Tool
Metrics has dominated the PPC arena but video testing measures the impact of search results and PPC ads on an emotional level. We watch user behaviors and even why competitors may draw more clicks. Testing landing pages and more importantly the conversion path, will further insure improved returns for the client.
The entire conversion path in most PPC campaigns needs to be tested and there is no better tool for immediate understanding than direct user video testing in the hands of an experienced consultant.
Video Testing Isn't THAT Expensive
It probably costs less than you think. Done in house, the software and hardware is under $3,000 assuming you have a good laptop. You do not need a lab, in fact in many cases it can be preferable to test in other environs. You often have to pay your test candidates from $25 to $50 per hour, more in the big cities.
If you run the (free) cost justification tool offered by Dr. Deborah Mayhew http://drdeb.vineyard.net , the cost of even outsourcing testing is negligible compared to the huge returns it typically generates.
Bottom line, it is hard to quantify what a consultant will charge for a video test, though the range of $1,000 for a quick test to $50,000 for a full blown large scale site test is probably realistic.
Data mining is an essential part of online marketing but in this competitive time, offering additional insights and information improves all our offerings. Good basic video testing is within the budget of the small business person today and the typical ROI for our clients is 2 months or less. We have customers who have reaped millions of dollars in returns. Leading companies combine data mining with direct user testing (Amazon built their business on it). Optimizing the User Experience is on everyone's lips these days and by combining data mining with direct user video testing, you will be offering your clients the best that is available and I guarantee that you will find the results will improve both your customers and your bottom line.
To learn more about direct user video testing, visit the download page to watch clips of the process
This article was inspired by a conversation with Dr. Gregory Piatetsky. Please check out his site, it is the best site on the web for information on Data Mining.
This article is free for reprint however you choose as long as you request permission, provide a link to this site and acknowledge ownership. If you need more information or wish to discuss any aspect of this policy or the article please feel free to email me. - Thanks, Todd
© 2008 Web Marketing Resources LLC Norwalk, Ct USA | services | facilities | testimonials |resources | site map | about us | contact us | home