#ATD2016 An Alternative to Kirkpatrick Evaluation

ATD International Conference Talent Strategy Mapping from Jon Ingham

I've come to ATD's main international conference after speaking at ATD MENA last year.  But also because I've been following the tweets from the last couple of years and have been increasingly concerned by the heavy focus on the Kirpatrick evaluation model here (much, much stronger than the CIPD's Learning and Development Show etc).

Kirkpatrick Partners suggest that "Don Kirkpatrick’s groundbreaking four levels of training evaluation is the most widely used training evaluation model in the world. Ask any group of trainers whether they rely on it in their practice, and you’ll get an enthusiastic affirmation."

The first statement is absolutely true.  The second is nonsense.  Come on!, we all know if doesn't work.

Last year there was also a session on Kaplan and Norton's Balanced Business Scorecard and how that helps training and learning evaluation.  It doesn't.  The BBS works for the rest of the business because it provides balance, flow and the ability to link planning objectives (the business strategy map) to measures (the scorecard).  It doesn't work for talent management and development because it doesn't give us balance or flow.

So I decided to come and talk about my own model, based upon the strategy map, which does provide these benefits (have a look here for an explanation).  It's also very closely related to Kirpatrick's 4 levels but because it's organic vs mechanistic (ie I don't tell you what you need to measure, this comes out of your own business and talent management / development strategy) it works whilst Kirkpatrick doesn't.

Participants in my session had an opportunity to try the approach out too and the strategy maps / value matrix they came up with for a health insurance business represented in the room is provided below (thanks Keith):

The objectives and measures in here are what I suggest WHA should evaluate, and the measures for evaluation are provided by the framework.  There's no need, worry, complexity or confusion to have to consult Kirkpatrick on what a particular measure should be.  And if it's not in the value matrix, you probably don't need to both evaluating it.

I asked our 'client' (Keith, the CHRO) whether the measures we'd identified were mainly the same or different from those they currently use for evaluation and he said they were largely the same.  That was a slight shame but participants had only had a maximum of 15 minutes to play with my framework, so I wasn't too disappointed.

And I think it does demonstrate the main point I was trying to make which is that Kirkpatrick's four levels are too constraining.  Eg up at the top of the matrix there's an objective about improved collaboration which is going to be measured through network mapping.  A great, perfectly aligned measure.  But where is that in Kirkpatrick's system?

It also demonstrates the difficulty measuring strategic attributes (those at the top of the framework) in an objective, quantified way.  Eg we didn't get to develop a measure for change agility, but I bet it won't be a percentage or rating, it's going to be something a bit more complex and therefore qualitative and potentially subjective, which is fine - but of course rules out doing a quantitative ROI study.

But I'm going to leave this post with my criticism of the Kirkpatrick model - if you want a critical review of Jack Phillips ROI methodology have a look here.


Later: actually I am going to extend my discussion into Phillips methodology as well.  This is because I've been reading more of his writing at the ATD conference and I think this demonstrates very effectively why Kirkpatrick / Phillips doesn't work.

Firstly, the article in the ATD's current TD magazine: The Value of Engagement.  Because Phillips uses the Kirkpatrick levels he's forced into completely inappropriate measures.  How do you measure engagement?  Not through 'reaction to engagement'!  What on earth is that?

Or take his new book on the ROI o Organisation Development.  There's not one single mention in the index of complexity, and only a couple of brief mentions of intangibles.  Ie there's no recognition at all that when you focus on something as deep as OD most of the outcomes you're creating fall into Phillips' bucket of intangibles and don't play a role in his perfectly precise ROI equation.  So what's the point of it?  I wasn't in the bookstore long enough to work out whether he recommends a 'reaction to organisation development' measure.  But wouldn't that be bizarre!

Actually, all you've got to do is to go back to the value matrix, check how you're planning to create engagement or a better (in WHA's case more agile) organisation and then measure / evaluate that.  Which is why we need a qualitative measure for change agilty.  And a discussion rather than a single reliable but completely invalid figure for ROI.


You can review the slides from my ATD presentation / workshop session here. 

• Consulting   Research  Speaking  Training  Writing
• Strategy  - Talent - Engagement  - Change and OD 
• Contact me to create more value for your business
• jon [dot] ingham [at] strategic [dash] hcm [dot] com