How not to write a case study

When you’re in marketing, and you see a case study headline like “How ‘This girl can’ got 1.6 million women exercising”, you get a bit excited.

I’m not ashamed to say a headline like that is like porn for us. Eager, hot statistics showing us the good stuff right upfront. Yes, baby. This is what we’re here for.

Unfortunately, that statistic is about as close to reality as a UKIP referendum leaflet.

Let’s cover the campaign itself before we dive into the stats. ‘This Girl Can’ was a campaign from Sport England and FCB Inferno; the agency responsible for such innovative ad campaigns as “buy savoury biscuits and meet Jessie J” and weird glob balls for Sky.  The first TV ad for ‘This Girl Can’ debuted on ITV on 12 January 2015, and after that the campaign took off pretty quickly with more TV ads, radio, billboards, display, Facebook videos and everything in between.

Full disclosure – I didn’t hate This Girl Can. And I hate everything.

Firstly, there was a legitimate problem to be solved here; a clear gender gap in sports participation in the UK. The campaign itself was really well presented campaign with strong messaging and a brilliant hook. The overall purpose of the campaign was conveyed cleverly in everything they produced, and the overall execution was actually pretty nice. Which is why I’m so angry that the results are being reported in such a bizarre way. This didn’t have to happen.

Based on the content of the case study, the video has been viewed 37 million times. That’s great. An awesome result that needs no fluffing at all.

Ah shit. They fluffed it.

I don’t need to tell you lot this, because you already know that total views does not equal unique views or unique viewers. So why make the leap? Was this just clumsy wording? A typo? Or did they really not understand that these are not the same thing? I’m actually kind of impressed that they managed to so directly contradict themselves in one neat, shareable graphic for Sport England. Efficient.

They also haven’t taken into account that the campaign – despite being run by Sport England – was rolled out across the whole of the UK. So those 37 million “of you” include Scottish, Welsh, and Northern Irish women. And men. Yep. Men saw this too. They don’t look away when ads aimed at ladies come on tv you know. They can even tough it out through ads for Tampax and Michael Bublé Christmas CDs.

Now the really problematic figure – 1.6 million. Where did that come from? I’m glad you asked. Let’s look at what FCB Inferno and Sport England tell us.

The increase in the number of women playing sport has driven an overall increase in the number of people regularly playing sport. This stood at 15.74m in the 12 months to the end of September 2015, up by 245,200 compared with the previous figures published in June.

So the only measurement taken in the number of people playing sport in England *since* the campaign went live – that we have data for – shows an increase of 245,200 in 2 months. That’s an impressive increase. But it’s also not broken down by gender. Even if we ignore the gender gap, and assume 50% of those people were women, that’s an extra 122,600 women in two months.

What the hell happened between January and June? Are they saying that 1.4 million women started playing sport during those months, then the rate of growth just fell off a cliff?

Overall, there has been an increase of 1.65m since London secured the rights to host the Olympic Games.

Ah cool, we have a longer term report now. That shows a total of 1.65m more people playing sport since, um, July 2005.

Maybe there’s some more research that I can look at? Oh, cool they mention the Sport England ‘Who Plays Sport?’ report right there in the case study. This report is the largest survey of sport and active recreation in Europe. And it’s been tracking figures continuously through the ‘Active People’ programme since 2005. This sounds like a great source of reliable data on who actually plays sport.

During the year up to April 2016, 15.8 million people aged 16 years and over in England played sport at least once a week. That’s an increase of 1.75 million since 2005/6.

So, 1.6m women have not started exercising because of the ad campaign. In the space of three sentences in the case study, they tell us it’s true, then directly contradict it, direct us to a study confirming it’s bollocks, and then invent another figure entirely.

I still couldn’t work out where this mythical 1.6 million figure came from, until I found another case study of the same campaign from January 2016.

Sport England announced yesterday that it has inspired 2.8 million women to do more physical activity (figures are based on a sample survey of 1,000 women, in which 40% said they had done more exercise as a direct result of seeing the campaign).

A survey of 1,000 women? What are they scaling up to? I’m so confused. That’s FAR too small a sample to make a national assumption on. Are they saying that all 1,000 women said they had recently started exercising? And that’s a huge number of people reporting an increase in exercise as a direct result of the ad.

Something isn’t right here. Where did this sample come from? What was the age/geography range? Where did they conduct the study? How did they get results that directly contradicted their own long term research programme? And why didn’t they question the results when there was such a direct contradiction?

Think about it – if this was a survey of women aged 16-25, entering a gym or fitness centre, then of course they’re going to see a high number of people positively responding that yes they are exercising. And they’re also more likely to have seen the campaign because they will probably have had to walk past three fucking posters with the slogan on it on the way in.

Also, guys, it looks like the number of tweets/active community members hadn’t changed between the first and second campaign review. Nothing in 4 months? That’s not good. Is everything okay over there? Or was this particular stat just not updated and you hoped no one would notice?

So. Why fudge the figures in the first place?

This was a decent campaign – with legitimately positive outcomes – that didn’t need any fudging. ‘This Girl Can’ got the type of press coverage most of us can only fantasise about, and created a genuine debate among women about the link between their perceptions of participation in sport and the reality of it. The ads resonated with women in a very real way, and portrayed us in the kind of body positive, objectification-free light that’s both rare and encouraging.

I just can’t understand why they invented figures when the real results were so tangible and positive.

Wait, this campaign cost £10 million?

Never mind. I’ve figured out why they fudged the figures.