When a worker uses generative AI to complete an unfamiliar task, the result may look like successful upskilling. But two different things can be happening: the person may be producing better work with a tool, or the person may be acquiring a skill they can later use without it. Those outcomes matter for different reasons, and organisations should not treat them as interchangeable.
A peer-reviewed study published in Nature Human Behaviour on 26 October 2026 offers a timely test of that distinction. In a randomized controlled trial involving 487 management consultants, access to ChatGPT and training in its use improved performance on data-science tasks outside participants’ existing skill sets. Yet the researchers found no evidence that the treated group had gained more technical knowledge when tested without ChatGPT after the experiment.
The finding is not that AI is useless for learning. It is that generative AI skill acquisition cannot be inferred from assisted performance alone. This article explains what the study establishes, what remains uncertain, and how employers and training teams can test whether AI-supported work leads to durable capability.
What the 487-worker trial actually tested
Wiles and colleagues recruited 487 management consultants and randomly assigned participants to a treatment or control condition. The treatment provided access to ChatGPT and training in how to use it to solve data-science problems beyond the consultants’ existing skill sets. The tasks were designed so that ChatGPT could not simply solve them independently; participants had to combine their own judgement with the tool.
The treated workers performed better than the control group on three tasks. However, they did not perform at the level of expert data scientists. When the researchers later assessed technical knowledge without ChatGPT, they found no evidence that the treatment group had become better at answering those technical questions.
The paper was published on 26 October 2026, but its methods report that the study data were collected in 2024. It is therefore a newly published analysis of an earlier experiment, not a direct test of the current model landscape. That timing matters when interpreting the result: model capabilities and workplace practices continue to change, while the underlying distinction between aided performance and retained skill remains important.
Read the primary source: Wiles et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (published 26 October 2026), DOI 10.1038/s41562-026-02566-6.
Performance with AI is not the same as learning
For workplace learning, it helps to distinguish three outcomes that are often collapsed into one.
1. Assisted task performance
This asks whether a worker can produce a useful output while an AI tool is available. It is a legitimate business outcome. A data analyst may be able to explore unfamiliar code, a project manager may structure a dataset, or a consultant may test a statistical approach with AI support. If the work is accurate, safe and efficient, the tool may expand what the person can accomplish today.
2. Capability while the tool remains available
This asks whether the worker can repeatedly complete a class of tasks by combining their own judgement with AI. The answer depends on more than model quality: it also depends on whether the person can recognise errors, provide relevant context, choose suitable methods and validate the result. A worker who accepts every plausible-looking answer may produce inconsistent work even when the tool is powerful.
3. Independent, retained skill
This asks whether the person can still explain the underlying concept, diagnose a new problem or perform essential steps when AI is unavailable. It requires evidence from a test or task that does not allow the system to supply the answer. The Nature study found no evidence of improved technical-question performance without ChatGPT after the experiment. That result does not prove that no participant learned anything; it means the trial did not demonstrate an independent knowledge gain on the measure used.
These outcomes can all be valuable, but they should be named separately in training plans and reports. A productivity gain should not automatically be marketed as a learning gain.
Why the distinction matters for employers
Organisations often invest in training because they need people to make reliable decisions, adapt to unfamiliar situations and transfer knowledge to new assignments. If AI helps an employee complete one task but the employee cannot recognise when the same method is inappropriate, the organisation may have gained short-term output without building the capability it expected.
The reverse is also possible: a programme may deliberately teach employees to use AI as part of their role rather than requiring them to reproduce every step unaided. That can be a sound objective. Many jobs are performed with tools, reference materials and colleagues. The key is to define which tasks require independent mastery and which require competent, supervised human–AI collaboration.
The distinction complements evidence from other workplace-training research. Prowad’s analysis of a 2,574-employee adaptive-training trial examined small knowledge gains from a specific training package; the new Nature study asks a different question about performance on unfamiliar tasks and knowledge without AI. See AI training at work: what the 2,574-employee trial found for the former evidence, and the guide to AI in vocational education and training for implications for skills systems.
What the study does—and does not—tell us
The randomized design strengthens the study’s ability to compare the treatment and control groups. The tasks also required human contribution rather than allowing ChatGPT to generate a complete solution on its own. This makes the result relevant to jobs in which people use AI to work beyond their current expertise.
Several limits still matter:
- One professional group: management consultants are not representative of every occupation, training level or workplace.
- Specific tasks and tool access: the intervention combined access to ChatGPT with training in how to use it. The result should not be read as an estimate for every AI product or course design.
- Knowledge is not the whole job: technical-question performance without AI is one indicator of learning, not a complete measure of collaboration, judgement, job quality or long-term transfer.
- Duration remains important: a post-experiment test cannot establish how capabilities develop over many months or how repeated practice changes results.
- Research transparency: the article discloses that several authors were or had been affiliated with BCG and that one author was an OpenAI employee; the study was supported through BCG employee time and internal resources. These disclosures do not invalidate the findings, but readers should consider them alongside the design and limitations.
The appropriate conclusion is narrow: in this experiment, AI access and training improved performance on the tested tasks, but the researchers did not find evidence of improved technical knowledge on the unaided post-experiment measure. Further research across occupations, tasks and longer follow-up periods is needed.
How to design AI-supported training that builds capability
Training teams can use the distinction between assisted performance and independent skill as a design principle. The aim is not to remove AI from every learning activity; it is to make sure the programme measures the outcomes it promises.
Step 1: Define the capability before choosing the tool
Write down what a learner should be able to do at the end of the programme. For example, “produce a dashboard with AI assistance” is different from “explain which data-quality checks are needed before a dashboard can be trusted.” Both may be useful, but they require different assessments.
Step 2: Record a baseline
Use a short, job-relevant task before training. Where the role requires independent competence, include an unaided component. Where the job is explicitly AI-assisted, assess whether participants can set the task up correctly, verify outputs, identify uncertainty and explain when human review is required.
Step 3: Teach verification, not just prompting
Ask learners to check assumptions, test outputs against trusted sources, inspect calculations or code, and explain why a result is fit for purpose. A polished answer is not evidence of correctness. This is especially important when an unfamiliar technical task could produce plausible but consequential errors.
Step 4: Fade support deliberately
Early practice can include demonstrations, hints and guided AI use. Later practice should gradually remove some of those supports. Learners might first solve a problem with AI, then critique an AI-generated solution, and finally complete a comparable task without the tool if unaided performance is part of the target. The sequence should match the job rather than follow a one-size-fits-all rule.
Step 5: Test transfer after a delay
Repeat a short assessment after several weeks and use a new problem rather than the exact exercise learners memorised. When appropriate, include supervised observation of real work. Immediate satisfaction and course completion can reveal engagement, but they cannot establish retention or transfer by themselves.
Step 6: Report separate outcomes
Use a simple scorecard with at least three columns: output quality with AI, independent knowledge or performance, and transfer to the job. Track time saved and error rates where those measures are meaningful. If AI improves output but not unaided skill, report that honestly and decide whether independent mastery is actually necessary for the role.
For a broader procurement process, use Prowad’s evidence checklist for evaluating AI learning tools. Although it was written for education settings, its core questions—what is measured, who benefits, how errors are handled and whether learning persists—also help workplace training teams.
A practical pilot for the next 30 days
A small pilot can answer more useful questions than a broad rollout based on vendor demonstrations.
- Week 1 — Set the standard: choose one skill, define acceptable performance and identify which parts must be completed independently.
- Week 2 — Run a controlled comparison: compare the AI-supported pathway with the current training approach where feasible, using comparable time and assessments. Document any differences between groups.
- Week 3 — Audit the work: have a qualified reviewer inspect a sample of outputs for accuracy, reasoning, safety and appropriate escalation. Record how often participants detect AI errors.
- Week 4 — Test without support and decide: use a fresh task to assess retention, compare the result with baseline, and decide whether the benefit justifies the costs and risks. Plan a later follow-up rather than treating the 30-day check as proof of long-term mastery.
For regulated or safety-critical work, do not use a short pilot as a substitute for formal qualification, supervision or required certification. AI-supported practice should complement those safeguards, not bypass them.
How this fits the wider workplace-learning evidence
The OECD’s 2026 report Developing Vocational Education and Training with Artificial Intelligence examines how AI may affect curricula, qualifications and occupational standards. That system-level view matters because a skill is not simply the ability to produce an output: it also involves standards, judgement and the ability to perform within a particular occupation.
A 2026 scoping review in Review of Education, “The adoption of AI technologies and data to support workplace learning in the corporate sector”, maps uses such as skill-gap analysis, personalisation, feedback and recommendations. A scoping review helps describe the field and identify gaps; it should not be mistaken for proof that every listed use causes better learning outcomes.
For educators and training leaders, the lesson is consistent with research on independent learning: assistance is most useful when people retain the ability to judge, verify and explain what they are doing. See also Prowad’s discussion of AI and independent learning.
Conclusion: measure the skill, not only the assisted result
The 487-worker trial published in Nature Human Behaviour shows why the phrase “AI upskilling” needs precision. ChatGPT helped management consultants perform better on unfamiliar data-science tasks, but the study did not find evidence that they answered technical questions better without the tool after the experiment. Better supported performance is valuable; it is simply not identical to demonstrated knowledge acquisition.
Before scaling an AI training programme, define the capability, measure a baseline, teach verification, fade support where appropriate, and test a new task after a delay. Report AI-assisted output and independent competence separately. That approach allows organisations to benefit from AI today while making more credible claims about what their people have actually learned.
References
- Wiles, E., Krayer, L., Abbadi, M. et al. (2026). Generative AI and the temporary upskilling of knowledge workers. Nature Human Behaviour. Published 26 October 2026. DOI: 10.1038/s41562-026-02566-6.
- OECD (2026). Developing Vocational Education and Training with Artificial Intelligence. OECD Publishing.
- Pham, P. et al. (2026). The adoption of AI technologies and data to support workplace learning in the corporate sector: A scoping review. Review of Education, 14, e70198.