Many CRO experts are hardcore believers in statistical significance. And I am too.
It’s fantastic to make data-driven business decisions based on real world evidence. Marketers guess too often.
But where the hardcore reliance on significance can let you down is when it becomes the driving factor in decision making. I see people holding out for a specific level of significance all the time and it distracts from the core mission – incremental growth.
The truth is, there’s nothing magical about 95% statistical significance, 99% or even 75%.
So when it comes time to deciding whether to end a test or keep it running to reach a magical significance level, it’s important to assess risk.
As CEO of New Republique, Nima Yassini writes, there is risk involved at every level of significance.
There’s also a massive cost in waiting.
Another factor to pair with risk appetite is opportunity cost. By delaying the end of one experiment by one month to seek 95% significance you are also losing the opportunity to run the next experiment on that page or other pages.
So waiting for that magical significance number can end up costing you more than what you were prepared to risk initially.
This is especially true for high traffic websites. Any delay in implementing your next test could really cost you.
So let’s not get caught up in reaching a certain level of significance for significance’s sake. It’s all about weighing up the risk and acknowledging the opportunity cost that comes with delay.
People often ask me:
When should I run a test?
My answer is usually:
If you can test, you should test.
Why? Testing gives you answers you won’t get any other way. Answers like:
- Does this variant convert customers better than what we had before?
- Does it keep customers active?
- Does it retain customers better?
- Does it improve average order value, average lifetime value?
- Does it grow our profit?
When deciding what to do next, try to avoid the urgent.
Urgent is rarely important. Urgent rarely brings long term value.
So ask yourself, what should I do next that will return dividends well into the future?
Like compound interest, these small, regular investments will accumulate over time and turn into something massive.
Don’t stop testing.
There’s no such thing as finished anymore. In business, your competitors will fly past you the moment you stop.
Anyone can launch a startup in your industry, this afternoon.
So if you stop moving, you’re going backwards.
The beauty of testing is that you can learn something new each day. It could be as simple as ‘that didn’t work’.
The trick is movement. Momentum.
That’s what keeps you in the game. Not every swing will be a home run. And that’s perfectly acceptable. What’s not acceptable is paralysis.
It’s more than running A/B tests.
It’s more than white lab coats.
More than reaching statistical significance.
It’s a state of mind. A worldview where you’re willing to fail.
A way of thinking that challenges the way you run your business. Experimentation means you’re willing to start from scratch and learn exactly what your customers want.
It trades everything you thought to be true for things you know to be true.
It never stops.
And it’s the only sustainable way to grow.
When going into any meeting, presentation or especially a challenging conversation, it really helps to have a framework to structure your thoughts.
Barbara Minto’s ‘Pyramid’ structure is one I’ve just learned about and it’s very effectvel.
It goes like this:
Set the context, environment and go through any supporting data. Generally give the lay of the land or the key facts so far.
What’s gone wrong? How are our plans threatened and what’s the potential impact? Explain how and why it’s got to this point.
There’s nothing better than a question that you already have the answer to. Summarise the complication with a question that prompts a resolution… and then give your pre-prepared answer.
State your proposed solution. Then, explain how you go there. List as many fact-based reasons you can as to why this answer is the best way forward. If someone shoots down one of your reasons, don’t panic! Keep going through the list like a ruthless prosecutor.
At the end, take any questions and address any concerns. If everyone is on board (they should be, because you overwhelmed them with reasons to be), allocate actions so that the solution can come to life.
Even if the four steps above don’t go to plan, at least you had one. Appearing in control is a victory in itself and will help you influence the outcome more often than not.
Success has just as much to do with communication as it does with talent.
Knowing who is doing what, in which order, while the project is in motion, is the key to achieving an outcome on time.
Talent, skill and hard work are known factors that impact quality.
Less stated, however, is the need to coordinate, communicate and plan.
Without these 3 things no one will ever see the finished product, which is a huge waste of effort.
Talent + skill + hard work = quality of product.
Coordination + communication + planning = delivery of product.
A lot of time can be saved by getting the people who actually do the work in the room early on.
Usually, time spent talking about possible challenges and solutions is completely wasted because no one thought to ask the person who will do it.
- What do we need to do
- Who does it
- When does it need to be done
But don’t ask:
- How do we do it
- What happens if we do it like this
- Why can’t we do this
…until you’ve got the right person answering the questions.
Have you ever felt that emotional, cloudy feeling in your brain when something seems so hard that you don’t know what to do next?
When the speaker makes the wrong assumption. Someone changes their mind. An agreement is broken.
Our mind can cope with only a few challenges at a time. In meetings, when more than 3 barriers are raised, notice how everyone’s confidence and constructiveness drops?
There’s a simple way to overcome this. Since our brains black out, it’s best to create a ‘parking lot’ on the whiteboard.
The parking lot is somewhere issues, blockers or cons can be stored and addressed at a later time. They are things raised during discussion, verbalised, noted and recorded.
Our brains love this. We don’t panic that they may be forgotten. We don’t stress about getting sidetracked. We don’t accuse people of hijacking the meeting.
When the time comes to go through the parking lot issues, you will find that the emotion has dispersed and once unpacked they are much easier to solve than previously thought.
The Parking lot technique is a simple way to keep meetings on track (and on time!) and to keep everyone focused on achieving the same thing. Try it!
A home run is an incredible way to win the game. You risk it all on one big swing, hoping that night will be the night you single-handedly make the difference.
However most teams in the MLB this season average less than 1 home run per game. Usually, games are won by singles or doubles, accumulated over an inning – not a magical moment of individual brilliance.
So it is with our work or any pursuit we are trying to excel in. Undoubtedly, we dream of a home run each time we wake up in the morning, hoping that today will be the day the crowd finally witnesses our hidden talent.
Though instead of aiming to hit it out of the park each time, we should try to simply hit the ball first. Then, hit singles. Hit singles as consistently and reliably as anyone. Turn up, deliver what was promised, contribute, improve, share. Hit singles every time.
Eventually, you might even hit a home run. Yet until you are known as the person who is reliable, the one-off remarkable success will only be viewed as a fluke.
True success comes from mastering the basics, over and over again. This is a skill as rare as smashing it into the crowd.
Image: Werner Kunz