Summary: Most organisations measure whether trainees liked a course; few measure whether results changed. A digital system for the Kirkpatrick model is mostly a measurement plan per programme, timed surveys and supervisor checks, reminders, one results database and a link to the operational KPIs. AI helps as a supporting text-analysis layer, with people reviewing every output. Sized honestly, that is three servers and one GPU, not an AI platform.
This month a government buyer asked the market for an “integrated digital system” to measure the impact of its training, with AI, dashboards and fifteen deliverables. Under the AI wording sits the Kirkpatrick model, digitised: plan the measurement for each programme, collect the evidence at the right moments, and link it to the results the organisation cares about.
The finding matters for anyone buying training evaluation software: most of the value is in the workflow and the KPI link, and most of the risk is in calling it an AI platform.
The Kirkpatrick model in one paragraph
The model measures training at four levels. Level 1, reaction: did trainees find it useful? Level 2, learning: did they gain the knowledge and skill? Level 3, behaviour: do they apply it on the job? Level 4, results: did the business outcome move? The current version, the New World Model, says to design backwards: start from the level 4 result, then decide what behaviour, learning and reaction must be measured to explain it. The buyer’s own document said the same, telling bidders to start “from the target results and then working back to the lower measurement levels”.
Why most organisations stop at level 1
Ninety-two per cent of organisations measure reaction, and few measure results (ASTD, 2009). Research on why points at time, managerial support and evaluation expertise (Kennedy and colleagues, 2014). The skill gap is as large as the software gap. What fixes it is a plan that tells each person what to collect, when, and why.
What training impact measurement software actually does
Strip away the labels and the system does seven things:
- Holds a measurement plan per programme: the target KPI, where its data lives, which instruments measure each level, and when.
- Sends timed surveys and knowledge checks to trainees, and behaviour checklists to their supervisors weeks later.
- Chases late responses with reminders and escalation.
- Stores every answer in one results database.
- Calculates the indicators for each level.
- Shows dashboards by programme, unit and period.
- Produces the reports management reads.
Nothing off the shelf models the first item well. Commercial evaluation suites are cloud-first and priced per learner. The Microsoft route of Forms, Power Automate and Power BI is cheapest, but needs a cloud tenancy cleared for the data. For a security-minded buyer, a custom plan engine on open components usually fits better: a form engine, a workflow engine for the reminders, PostgreSQL for results and a reporting layer on top.
Where AI fits, and where it does not
The buyer’s document was honest about this: AI “will serve as a supporting analysis tool, with interpretive and managerial decisions remaining under human review”. In practice that is batch analysis of free-text survey comments: themes, sentiment, the barriers trainees name when they cannot apply what they learned. Every output goes to a person before it reaches a report.
The volume is modest. An organisation running tens to low hundreds of programmes a year produces low thousands of survey answers a month. Three virtual machines and one GPU server for a local language model cover it with room to spare. Anyone quoting a large AI platform for this is quoting the label.
Level 4 lives in operational systems
The hardest part is not software. Level 4 results sit in the systems where work happens: incident logs, quality audits, drill results, response times. HR systems rarely hold them. So the first discovery question for any project is simple: which system holds each target KPI, and who can give read access to it?
Training that already produces data helps. Our VR training systems record what the trainee reviewed and marked in each scenario, with eye tracking and heart rate for the debrief. That is level 2 and level 3 evidence a survey cannot give, and it can feed the same results database. The argument for building VR training around measurement is in VR training does not beat the classroom. The method does.
What to ask before you buy
- Does the system hold a measurement plan per programme, with the target KPI named?
- Which operational systems will it read for level 4, and who has agreed to give access?
- How are supervisors prompted for level 3, and what happens when they do not answer?
- Where does the AI run, and who reviews its output before a report?
Frequently asked questions
What are the four levels of the Kirkpatrick model?
Reaction, learning, behaviour and results. They measure whether trainees found the training useful, learned from it, apply it on the job, and whether the business outcome changed.
Where should training evaluation start?
At level 4. Name the business result first, then decide which behaviour, learning and reaction measures explain it. That is the Kirkpatrick New World Model.
Does training impact measurement need AI?
Only as a support. AI is useful for analysing free-text comments at scale, with people reviewing the output. The core is the measurement plan, the timed data collection and the link to operational KPIs.
What data proves level 4 results?
Operational records: incident logs, quality audits, drill results, response times. They usually sit outside HR, so identifying the owner of each system is the first step.
Next step
If you need to show what your training changes, we can design the measurement plan and the system behind it, and connect your VR training data. See our VR training simulators, or get in touch.
