Hypothesis Sketching with Craig Sullivan

One of the biggest problems in split testing is the fact that the hypothesis for most tests is weak, unsupported or driven entirely by the diseased imagination of somebody who is very senior but unforgivably biased. In his workshop, Craig Sullivan discussed how to ensure the right inputs are delivered for the Hypothesis Design stage and led participants in a practical Hypothesis Sketching exercise to come up with outline wireframes for testing.

The hypothesis is the crux at the centre of a lot of our work. Every time you make a change to the design of a product it is based on an informed guess – a hypothesis.

The hypothesis is one of the places people can wrong with AB testing, so it is crucial to get it right.

A hypothesis is a proposed explanation for a phenomenon. The scientific method requires that one can test it. This means you should collect data that tells us if we have failed.

Craig summarised a good hypothesis as:

“Because we observe data A and feedback B, we believe that doing C with People D, will cause outcome E, we will know this when we observe data and obtain feedback.”

He stressed that you need to decide your measurement metrics before you conduct your experiment.


Craig provided a real client website from a client that has asked for AB testing. However, the client has done no market research or usability testing previously. They would like to AB test, but it would take nine years to run based on their current rates of traffic. The targets for testing are the home page and the up sell mechanic, as these can be modified fairly quickly.

Craig challenged participants to re-sketch one of these two elements of the site on their own and to think about why they are changing these elements. He then asked everyone to present their ideas in groups of five to expose the ideas that went into each sketch.

Following this, Craig asked participants to group sketch on a larger piece of paper to combine the best of these ideas.


Problems highlighted within the sample website included:

  • The absence of a primary navigation
  • The need for key benefits/what they do to be clearer
  • The product strap line (explaining the point of the site) is buried

Suggestions to improve the sample website included:

  • Showing a sample report to clearly show the product offering
  • Reorganising website elements to bring key information further up the home page
  • Adding clear navigation


Craig emphasised the artificial nature of doing this exercise under such tight timescales. He outlined the real process, including gathering inputs from session replays, analytics, usability testing, voice of customers reports, service interviews, sales team interviews, surveys, cross device QA testing, social network analysis, competitor analysis and any other data they can get to inform a hypothesis sketching session.

Based on this evidence, he would then ask everyone to sketch their ideas and combine these using the pattern demonstrated by the workshop exercise so that design meetings are not dominated by one individual and all ideas can be explored.

Craig concluded by stressing that UX research with data and testing is a vital starting point. This gives you motivation and helps clients to evolve – throwing away ideas that aren’t working and improve those who do. Variation and the heritability of good ideas are key. Qualitative insight and quantitate data are vital to achieve these.

About Craig Sullivan

Craig has been blending UX, Analytics, Split Testing, Voice of Customer and Conversion Optimisation techniques for over 12 years. He’s also been building teams, launching products and hacking the growth of websites for companies like Google, LOVEFiLM, Lego, John Lewis, eBay and more.

He’s been bringing together the often isolated worlds of qual and quant measurement within organisations – to integrate a variety of discovery and inspection techniques and use these to prioritise, fix, improve or split test hypotheses about product design.

You can find him tweeting from @OptimiseOrDie on everything to do with Numbers, UX and Psychology. Although Craig confesses to being a shameless hacker of numbers, growth and marketing initiatives – his lifelong love has been in the boundary region between customers heads and the interface with products and services.

Follow Craig on Twitter: @OptimiseOrDie