When a worker uses generative AI to complete an unfamiliar task, the result may look like successful upskilling. But two different things can be happening: the person may be producing better work with a tool, or the person may be acquiring a skill they can later use without it. Those outcomes matter for different reasons, and organisations should not treat them as interchangeable.<\/p>\n
A peer-reviewed study published in Nature Human Behaviour<\/em> on 26 October 2026 offers a timely test of that distinction. In a randomized controlled trial involving 487 management consultants, access to ChatGPT and training in its use improved performance on data-science tasks outside participants’ existing skill sets. Yet the researchers found no evidence that the treated group had gained more technical knowledge when tested without ChatGPT after the experiment.<\/p>\n The finding is not that AI is useless for learning. It is that generative AI skill acquisition cannot be inferred from assisted performance alone<\/strong>. This article explains what the study establishes, what remains uncertain, and how employers and training teams can test whether AI-supported work leads to durable capability.<\/p>\n Wiles and colleagues recruited 487 management consultants and randomly assigned participants to a treatment or control condition. The treatment provided access to ChatGPT and training in how to use it to solve data-science problems beyond the consultants’ existing skill sets. The tasks were designed so that ChatGPT could not simply solve them independently; participants had to combine their own judgement with the tool.<\/p>\n The treated workers performed better than the control group on three tasks. However, they did not perform at the level of expert data scientists. When the researchers later assessed technical knowledge without ChatGPT, they found no evidence that the treatment group had become better at answering those technical questions.<\/p>\n The paper was published on 26 October 2026, but its methods report that the study data were collected in 2024. It is therefore a newly published analysis of an earlier experiment, not a direct test of the current model landscape. That timing matters when interpreting the result: model capabilities and workplace practices continue to change, while the underlying distinction between aided performance and retained skill remains important.<\/p>\n Read the primary source: Wiles et al., \u201cGenerative AI and the temporary upskilling of knowledge workers\u201d<\/a>, Nature Human Behaviour<\/em> (published 26 October 2026), DOI 10.1038\/s41562-026-02566-6<\/a>.<\/p>\n For workplace learning, it helps to distinguish three outcomes that are often collapsed into one.<\/p>\n This asks whether a worker can produce a useful output while an AI tool is available. It is a legitimate business outcome. A data analyst may be able to explore unfamiliar code, a project manager may structure a dataset, or a consultant may test a statistical approach with AI support. If the work is accurate, safe and efficient, the tool may expand what the person can accomplish today.<\/p>\n This asks whether the worker can repeatedly complete a class of tasks by combining their own judgement with AI. The answer depends on more than model quality: it also depends on whether the person can recognise errors, provide relevant context, choose suitable methods and validate the result. A worker who accepts every plausible-looking answer may produce inconsistent work even when the tool is powerful.<\/p>\n This asks whether the person can still explain the underlying concept, diagnose a new problem or perform essential steps when AI is unavailable. It requires evidence from a test or task that does not allow the system to supply the answer. The Nature study found no evidence of improved technical-question performance without ChatGPT after the experiment. That result does not prove that no participant learned anything; it means the trial did not demonstrate an independent knowledge gain on the measure used.<\/p>\n These outcomes can all be valuable, but they should be named separately in training plans and reports. A productivity gain should not automatically be marketed as a learning gain.<\/p>\n Organisations often invest in training because they need people to make reliable decisions, adapt to unfamiliar situations and transfer knowledge to new assignments. If AI helps an employee complete one task but the employee cannot recognise when the same method is inappropriate, the organisation may have gained short-term output without building the capability it expected.<\/p>\n The reverse is also possible: a programme may deliberately teach employees to use AI as part of their role rather than requiring them to reproduce every step unaided. That can be a sound objective. Many jobs are performed with tools, reference materials and colleagues. The key is to define which tasks require independent mastery and which require competent, supervised human\u2013AI collaboration.<\/p>\n The distinction complements evidence from other workplace-training research. Prowad’s analysis of a 2,574-employee adaptive-training trial examined small knowledge gains from a specific training package; the new Nature study asks a different question about performance on unfamiliar tasks and knowledge without AI. See AI training at work: what the 2,574-employee trial found<\/a> for the former evidence, and the guide to AI in vocational education and training<\/a> for implications for skills systems.<\/p>\n The randomized design strengthens the study’s ability to compare the treatment and control groups. The tasks also required human contribution rather than allowing ChatGPT to generate a complete solution on its own. This makes the result relevant to jobs in which people use AI to work beyond their current expertise.<\/p>\n Several limits still matter:<\/p>\n The appropriate conclusion is narrow: in this experiment, AI access and training improved performance on the tested tasks, but the researchers did not find evidence of improved technical knowledge on the unaided post-experiment measure. Further research across occupations, tasks and longer follow-up periods is needed.<\/p>\n Training teams can use the distinction between assisted performance and independent skill as a design principle. The aim is not to remove AI from every learning activity; it is to make sure the programme measures the outcomes it promises.<\/p>\n Write down what a learner should be able to do at the end of the programme. For example, \u201cproduce a dashboard with AI assistance\u201d is different from \u201cexplain which data-quality checks are needed before a dashboard can be trusted.\u201d Both may be useful, but they require different assessments.<\/p>\n Use a short, job-relevant task before training. Where the role requires independent competence, include an unaided component. Where the job is explicitly AI-assisted, assess whether participants can set the task up correctly, verify outputs, identify uncertainty and explain when human review is required.<\/p>\n Ask learners to check assumptions, test outputs against trusted sources, inspect calculations or code, and explain why a result is fit for purpose. A polished answer is not evidence of correctness. This is especially important when an unfamiliar technical task could produce plausible but consequential errors.<\/p>\n Early practice can include demonstrations, hints and guided AI use. Later practice should gradually remove some of those supports. Learners might first solve a problem with AI, then critique an AI-generated solution, and finally complete a comparable task without the tool if unaided performance is part of the target. The sequence should match the job rather than follow a one-size-fits-all rule.<\/p>\n Repeat a short assessment after several weeks and use a new problem rather than the exact exercise learners memorised. When appropriate, include supervised observation of real work. Immediate satisfaction and course completion can reveal engagement, but they cannot establish retention or transfer by themselves.<\/p>\n Use a simple scorecard with at least three columns: output quality with AI, independent knowledge or performance, and transfer to the job. Track time saved and error rates where those measures are meaningful. If AI improves output but not unaided skill, report that honestly and decide whether independent mastery is actually necessary for the role.<\/p>\n For a broader procurement process, use Prowad’s evidence checklist for evaluating AI learning tools<\/a>. Although it was written for education settings, its core questions\u2014what is measured, who benefits, how errors are handled and whether learning persists\u2014also help workplace training teams.<\/p>\n A small pilot can answer more useful questions than a broad rollout based on vendor demonstrations.<\/p>\n For regulated or safety-critical work, do not use a short pilot as a substitute for formal qualification, supervision or required certification. AI-supported practice should complement those safeguards, not bypass them.<\/p>\n The OECD’s 2026 report Developing Vocational Education and Training with Artificial Intelligence<\/em><\/a> examines how AI may affect curricula, qualifications and occupational standards. That system-level view matters because a skill is not simply the ability to produce an output: it also involves standards, judgement and the ability to perform within a particular occupation.<\/p>\n A 2026 scoping review in Review of Education<\/em>, \u201cThe adoption of AI technologies and data to support workplace learning in the corporate sector\u201d<\/a>, maps uses such as skill-gap analysis, personalisation, feedback and recommendations. A scoping review helps describe the field and identify gaps; it should not be mistaken for proof that every listed use causes better learning outcomes.<\/p>\n For educators and training leaders, the lesson is consistent with research on independent learning: assistance is most useful when people retain the ability to judge, verify and explain what they are doing. See also Prowad’s discussion of AI and independent learning<\/a>.<\/p>\n The 487-worker trial published in Nature Human Behaviour<\/em> shows why the phrase \u201cAI upskilling\u201d needs precision. ChatGPT helped management consultants perform better on unfamiliar data-science tasks, but the study did not find evidence that they answered technical questions better without the tool after the experiment. Better supported performance is valuable; it is simply not identical to demonstrated knowledge acquisition.<\/p>\n Before scaling an AI training programme, define the capability, measure a baseline, teach verification, fade support where appropriate, and test a new task after a delay. Report AI-assisted output and independent competence separately. That approach allows organisations to benefit from AI today while making more credible claims about what their people have actually learned.<\/p>\n A randomized trial of 487 management consultants found that ChatGPT improved performance on unfamiliar tasks, but did not demonstrate better unaided technical knowledge. Learn how to measure AI-assisted work and durable skill acquisition separately.<\/p>","protected":false},"author":1,"featured_media":404,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[24],"tags":[],"class_list":["post-403","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-learning-training"],"yoast_head":"\nWhat the 487-worker trial actually tested<\/h2>\n
Performance with AI is not the same as learning<\/h2>\n
1. Assisted task performance<\/h3>\n
2. Capability while the tool remains available<\/h3>\n
3. Independent, retained skill<\/h3>\n
Why the distinction matters for employers<\/h2>\n
What the study does\u2014and does not\u2014tell us<\/h2>\n
\n
How to design AI-supported training that builds capability<\/h2>\n
Step 1: Define the capability before choosing the tool<\/h3>\n
Step 2: Record a baseline<\/h3>\n
Step 3: Teach verification, not just prompting<\/h3>\n
Step 4: Fade support deliberately<\/h3>\n
Step 5: Test transfer after a delay<\/h3>\n
Step 6: Report separate outcomes<\/h3>\n
A practical pilot for the next 30 days<\/h2>\n
\n
How this fits the wider workplace-learning evidence<\/h2>\n
Conclusion: measure the skill, not only the assisted result<\/h2>\n
References<\/h2>\n
\n