Evidence based management and the dangers in surveys ~ Strategic Human Capital Management (HCM) Blog

Thursday 28 May 2015

Evidence based management and the dangers in surveys

I was a bit over critical of KPMG's evidence based HR report yesterday but it really did remind me of the sort of surveys reviewed on 'Bad PR':

“Getting a new job is so hard, you’ll need professional help!” says recruitment company

“Spending time in the bathroom together could save your marriage!” says bathroom retailer

“Women who diet via other methods are boring!” says diet drink company

"Big data and analytics are in" says big technology oriented consultancy?

There are a couple of common tricks of the trade in these kinds of surveys. One is picking the people who will give you the required answers.**

The second is asking questions in a way that will get you the most useful responses. At its most artful this is about nudging people towards a more desired output, for example by making 'yes' support your particular objective ("yes, I want to remain part of the UE", not "no, I want to remain part of the UK")

At it's most crass, it's about asking forced response questions which lever people into providing answers they don't mean. Management Today had a great article on this earlier in the year:

"How do companies such as OnePoll obtain such consistently scandalous, clickbait-friendly results? One favourite technique is the forced choice. For example, questions such as, ‘Who do you prefer to work for, men or women?’ where the only answers possible are ‘men’ or ‘women’. The logical responses - ‘I don't discriminate’ or ‘I don't know’ are not options — and respondents are forced to choose in order to claim their precious 10p reward.

A leading question in the same survey asks, ‘If you prefer to work for a man, why do you think women make bad bosses?’ with answers including ‘no time of the month’ and ‘easier to reason with’. Therefore whatever the survey result, the PR firm has a juicy headline to get their client's name into the papers followed by ridiculous, fabricated, pre-determined justifications for the nonsense headline. This is precisely how the Daily Mail's headline: ‘Men are the best bosses: Women at the top are just too moody (and it's women themselves who say so)’ was made. The story also featured in The Guardian, The BBC, on Five Live, The Wright Stuff and Loose Women. The somewhat less headline-worthy truth of the matter was never given a chance. That clearly wasn't the objective of the recruitment agency that funded the sexist survey."

A master in this type of research is Heathrow Airport, whose 'independent' 'community group' Back Heathrow generates some of the most biased research around.

Their latest resident survey asks 'if you knew that Heathrow would definitely close without expansion, are you in favour or against it expanding?' It's a compulsory question and respondents are forced into being in favour or expansion or of Heathrow closing* - despite there being no prospect of this at all since Boris island was kicked out of consideration. But the question doesn't even bother asking 'would you be' but simply asks 'are' as if the closure of Heathrow post a decision to expand Gatwick is already a done deal. You can immediately see the type of headline they're hoping to feed the Daily Mail from this.

Of course you can go even further and say that anyone responding to your survey is a supporter regardless of whether they respond positively or not (as seems to have happened in regard to Heathrow's survey in at least some cases too.)

It's not the worst kind of corporate misbehaviour we've experienced over the last few decades but it doesn't do anyone other than the Daily Mail any favours and adds to the need to be very cautious about the validity of research based evidence.

* = (or that they don't know what they think)

** = I'm not criticising KPMG's report for either of these 'tricks' - as I wrote yesterday I think they just over-extrapolated from the research findings. But then, I think we have to recognise that although Jonathan Ferrar from IBM, SAS, the Information Services Director at Unilever, John Boudreau, Anthony Hesketh etc all know their stuff, they will also provide a certain type of response.