#atdMENA Jack Phillips and the ROI of Talent Development in Saudi Arabia ~ Strategic Human Capital Management (HCM) Blog

Monday 16 March 2015

#atdMENA Jack Phillips and the ROI of Talent Development in Saudi Arabia

I'm not a fan of calculating ROI as I think there are plenty of other probably better things to spend time on. However when I'm pushed I usually recommend Jack Phillips approach as the way to do it.

I think that after seeing Jack present at ATD MENA I'm now even more strongly of my previous opinion on both of these points.

Why I recommend the approach - if you've got to do it!

Firstly, the approach does make a lot of sense. CEOs do want to see ROIs as well as analytics and it makes sense to respond with what they want when we can. So it's no surprise that whilst only 28% of organisations currently try to measure ROI (not many succeed!), 68% intend to do so.

I also like Jack's definition of value (see the slide above) which underpins his approach i.e. qualitative and quantitate, financial and non-financial etc.

I also think that Jack has identified the right levels of evaluation - levels 1 to 4 from Kirkpatrick's model of evaluation, with ROI as an additional level 5 and a 'new' level - level 0 for inputs. Actually you can see that these levels mark the same points as the steps in my value chain (incidentally, I've always had an 'input' level):

I also like the idea in Jack's V or Alignment model that Talent Development practitioners need to clarify their objectives at each of these levels before they think about evaluating at each of them:

And I like Jack's research into why companies choose to evaluate at levels 4 and 5, particularly the focus on importance in terms of link to strategic objectives:

I'm even fairly comfortable about the ROI methodology, though I do have one fairly large concern that I'll come to below:

But why in many cases you really shouldn't want to

So, on to my concerns,

Firstly, I still think the overall model is limited because it's so fixed. Yes, application is an example of an outcome objective but it's not the only thing you might want to focus on, or measure, at this step in the value chain. In particular, the model completely missed the role of human, organisational and social capital, which as I talked about in my session at ATD MENA, are actually the most important things we should focus on in HR and Talent Development.

Also, I still don't like the idea that you should identify objectives for a learning programme in order to evaluate it in Jack's V model ("the challenge is to connect learning to these measures.") The real thing you need to do is to develop good objectives and measures before you construct the learning. Doing this also ensures that you pick the right kind of objectives and measures and won't simply end up measuring 'application' because it's in Jack's model.

Apart from these points I can see how the model would apply to something like Jenny's sales training. But I still worry that the more strategic something is, the less likely it is to be measurable, and therefore mostly ends up in the intangibles bucket. In which case is it really worth the effort in going through Jack's process, and particularly the effort involved in isolating effects of the programme and converting some of the more tangible benefits to monetary value???

I asked Jack about this at the end of the programme and was pleased Jack seemed to agree with me that it wouldn't be. He gave me two examples, one where a company didn't do any evaluation and one where they did :

Firstly, Jack's ROI Institute has been working with EDF Energy. They've been putting a few tens of thousands of people through 3 hour sessions to support culture change - which has been a huge investment. So they asked Jack how they should evaluate it. But most of the benefits are intangible and would be hard to link to the development activity. Jack's advice was to decide not to evaluate it - but to communicate this decision in advance to stop problems cropping up later on.

There's another organisation which has been putting its leaders through the Advanced Management Programme at Harvard for the last 5 years but they've not been doing any evaluation. Again the outcomes are intangible so Jack got them ask participants to bring along a project then tracked participants' success in implementing that, using their new skills.

I quite like the first example. Some people might criticise the company suggesting that it shouldn't have done the development if it wasn't able to measure the impact. I'm glad that Jack agreed that this shouldn't be something which would stop them doing it. (The fact that there are much smarter ways of changing culture would be a much better reason for not doing it!)

I'm less compelled by the second example. Yes this company can now do some evaluation but actually they've completely changed the nature of their programme so they're not evaluating what was happening before. Also whatever business results participants produce are much more likely to be a result of as more discipled and measured approach to business improvement than they are the learning that participants had to support this. The ROI from the learning will only be a small part of the ROI of implementing the business projects but it will also be bigger than that small part as the execs will hopefully be able to solve other business problems too. So Jack may have got a number from the evaluation but it doesn't really correspond to the value of the development that's taken place.

Finally there's Jack's findings that more organisations intend to invest in calculating ROI in the future:

It certainly looks impressive that the proportion of organisations measuring ROI is due to leap up from 4 to 74%. But even more intend to focus on measuring business impacts - up from 8 to 96%.

Yet, this is 60 years after Donald Kirkpatrick first tried to get organisations to evaluate at level 4. If the proportion that do is still only 8% after all this time, I think 'the future' is some considerable time off!

There's a lot of talk about ROI and analytics and both Jack's and Jenny's approaches have utility and value in certain contexts. But we shouldn't pretend that we're going to start doing this outside these limited contexts. It's just not going to happen. So we need to find other ways of articulating the value provided by HR and Talent Development instead.

Some of this goes back to Jack's definition of value - we need to be able to measure qualitatively as well as quantitatively, and non-financially as well as financially. It's these areas which I think provide the greatest opportunities to improve.