Did the AI training work? How L&D leaders measure what stuck

The easiest numbers to collect are the ones that mislead

After a training day, most L&D dashboards fill up with two numbers: how many people completed the session and how they rated it. Both feel like evidence. A full room and a wall of five-star feedback do tell you the day ran well and people had a good time. They don't tell you whether anyone works differently the following Monday.

None of this is new. Learning teams have tracked attendance and satisfaction for decades, because those numbers are sitting right there at the end of a session. The trouble is that AI training is bought to change behaviour. Judge it on the two numbers furthest from behaviour and good programmes get written off as failures while bad ones get renewed. Measurement should be part of the buying decision too. It's one of the four questions in our buyer's checklist.

You can't measure change you didn't baseline

The most common reason AI training can't be evaluated is that nobody recorded the starting point. Six weeks later someone asks whether it worked. There's nothing to compare against, so the answer falls back to the completion rate and the happy sheet.

Baselining doesn't need to be elaborate. If you're working out how to roll out AI across your team, make the baseline part of that plan. Before anyone is trained, capture a few signals of how the team works now:

• How many people use an AI tool on real work in a given week, and what for. In most teams it's a small minority, and the baseline should say so.
• For two or three tasks the training is meant to affect, how long each one takes today and how it gets done.
• A short, ungraded confidence check. Can people describe when to trust a model's output and when not to?
• A couple of pieces of work from before the training, kept as a record of what "normal" output looked like.

None of this is precise science, and it doesn't need to be. You just need something real from before, so any claim about after has a comparison behind it.

The four levels, in plain English

The most durable framework for this is still Donald Kirkpatrick's four levels of training evaluation. It was published in the 1950s and is maintained today by Kirkpatrick Partners. It sorts measurement into four levels, each more useful than the one below. It's worth knowing precisely because most programmes only ever measure the bottom two.

• Level 1, reaction: did people find it engaging and relevant? This is the happy sheet. It's good for spotting a badly run session and useless as proof of impact.
• Level 2, learning: did knowledge or skill increase? A short before-and-after exercise on real tasks will show you. It beats reaction, but it still isn't the goal.
• Level 3, behaviour: are people doing the work differently weeks later? This is where the value is. It's also where measurement usually stops, because it's harder.
• Level 4, results: did the business outcome move? Think time saved, better quality or a backlog cleared. It's the hardest level to attribute cleanly, and it's the one leadership cares about.

Levels three and four are hard to measure. Any provider who promises a tidy number for level four before they've seen your work is overselling. Don't chase a perfect figure. Keep pulling your attention up from levels one and two, where the easy numbers live, towards the levels that reflect what you were buying.

What behaviour change looks like

Level three sounds abstract until you decide what you'd physically look for. These are the signals worth watching a few weeks after training, and you can see most of them without a survey:

• Are people reaching for AI on their own tasks without being told to? Can they say why they used it here and not there?
• Time to proficiency: how quickly someone new to a tool becomes properly productive with it, rather than just able to open it.
• Is the output better, or just more of it? More drafts is easy and often worthless. You want work that's as good or better, produced faster, by someone who can still stand behind it.
• Do people know the limits? Watch for someone catching a confident wrong answer, checking a fact at source or knowing what not to hand to a model. This is the skill that separates useful adopters from risky ones.
• Are trained people teaching sideways? When someone shows a colleague a workflow, the skill has probably stuck.

You won't capture all of this in a dashboard, so don't pretend to. A fifteen-minute conversation with three people who did the training, four weeks on, tells you more about level three than any automated report. Ask them what they changed. If they struggle to name anything specific, that's your answer, and it's a more honest one than a 92% satisfaction score. The people already showing colleagues their workflows are natural candidates for an AI champions network.

What we watched for on a real six-week programme

We delivered a six-week, hands-on AI programme for product managers at a Fortune 500 tech company, with a training partner. Because it ran over six weeks rather than a single day, we could watch behaviour change as it happened instead of guessing at it afterwards. That's the window levels three and four need.

We weren't looking at test scores. We watched whether people brought their own work to each session, and whether what they built one week was still in use the next. We also watched whether the questions got sharper, from "how do I prompt this" early on to "where does this fit in our process" later.

A single day is very hard to evaluate for behaviour change, because there's no "weeks later" built into it. The same gap is part of why one-off workshops don't stick. If measuring impact matters to you, the shape of the programme (spaced sessions, real work, a follow-up) matters as much as the content. That shape is what makes level three observable at all. We weigh up the formats in on-site vs live online training.

A measurement plan you can run

You don't need an analytics team to do this well. A workable plan fits on one page:

• Before: baseline three or four things (usage, task time, confidence, sample outputs). Write down what "better" would look like in concrete terms.
• During or straight after: a short applied exercise on real work for level two. Skip the multiple-choice quiz.
• Four to six weeks after: a light-touch check on level three. Use usage data if you have it, plus a few conversations about what changed.
• Later: go back to the one or two business outcomes you named up front. Attribution will be imperfect, so say so.

Keep it in proportion to the spend. Nobody needs a research paper. You need to know whether the money changed how the team works, so the next decision rests on evidence, not the last vendor's satisfaction slide. For the wider question of programme ROI, see how to measure AI ROI without fooling yourself. If you want training designed to be measured this way, that's what our AI training is scoped around. Talk it through with us.