Beyond Prompting: How Can Schools Measure Real AI Literacy?

As generative AI becomes part of everyday study, a practical question is becoming harder for schools and universities to avoid: how can educators tell whether students are genuinely AI-literate? A student may feel confident using ChatGPT, produce a polished assignment with it, and still struggle to verify a claim, recognize a fabricated reference, decide when AI assistance is inappropriate, or explain the reasoning behind a submitted answer.

This distinction matters because AI literacy is not simply familiarity with a chatbot or the ability to write an effective prompt. It includes understanding what a system can and cannot do, judging the quality of its output, making responsible decisions about its use, and retaining responsibility for learning. Recent research on performance-based testing and assessment-specific AI literacy suggests that education needs more precise ways to measure these capabilities.

This article reviews evidence on AI literacy assessment, considers what current measurement tools can and cannot tell us, and proposes practical principles for educators. It is an evidence-based synthesis of published research, not a report of a new experiment.

The measurement problem: confidence is not competence

Many AI literacy questionnaires ask learners to rate their own knowledge or confidence. Self-reports are useful: they can reveal attitudes, anxiety, perceived ability, and whether students believe they need support. But they do not directly demonstrate that a learner can perform a task successfully.

A systematic review in npj Science of Learning found that, although available AI literacy scales generally showed promising structural validity and internal consistency, relatively few had been tested for content validity, construct validity, responsiveness, or cross-cultural validity. The review also identified far fewer performance-based measures than self-report instruments. This points to a measurement gap, not proof that self-report tools are useless.

A later systematic review in Computers and Education: Artificial Intelligence examined 31 empirical studies using six AI literacy instruments across 14 countries. It found that AI literacy was associated with factors such as self-efficacy, attitudes, motivation, and digital competencies. It also reported differences between relationships measured with self-report scales and those measured with a performance-based test, raising the possibility that self-perceptions can distort how literacy is estimated.

The educational implication is straightforward: a survey can tell a teacher what students think they can do, but a performance task can provide evidence of what they actually do in a defined situation. Strong assessment systems should use each method for the questions it can answer. This distinction builds on the broader question explored in our guide to what AI literacy actually means and complements our analysis of the limits of AI detectors: neither confidence nor a detector score alone can establish what a student understands.

What performance-based assessment adds

One important contribution is the Generative AI Literacy Assessment Test (GLAT), reported in Computers and Education: Artificial Intelligence in December 2025. The researchers developed a 20-item multiple-choice test and evaluated it with 355 higher education students. They reported acceptable reliability and structural validity; GLAT scores also predicted performance on GenAI-supported tasks better than self-ratings such as perceived ChatGPT proficiency.

The finding supports a shift from asking only “How confident are you?” to also asking students to demonstrate knowledge and judgment. However, the evidence should be interpreted within its limits. The study involved a particular instrument and a sample of higher education students. A 20-item test cannot capture every real-world AI skill, and further validation is needed across age groups, languages, disciplines, and changing AI tools.

Performance-based assessment does not have to mean a long, standardized examination. A teacher might ask students to compare two AI-generated explanations, identify unsupported claims, verify a reference, explain why one answer is stronger, or revise an output while documenting what they changed and why. Such tasks make the underlying reasoning more visible than a confidence rating alone.

A second development: assessing AI literacy within academic assessment

A study published in Frontiers in Education on 10 September 2026 developed and validated the Generative AI Assessment Literacy Scale (GAA-LS) for higher education students. Rather than measuring general AI knowledge, it focuses on decisions students face when AI is used in assessed work.

The researchers began with an item pool and used expert review and pilot testing before analysing data from two student samples: 486 students in the first study and 798 in the second. The final 18-item instrument supported five dimensions:

• Assessment criteria awareness: understanding what the assignment requires and how work will be judged.
• AI-task appropriateness: deciding when AI assistance is suitable or unsuitable for a particular task.
• Verification and evidence checking: checking claims, references, and reasoning instead of accepting outputs at face value.
• Ethical attribution and academic integrity: following disclosure rules and taking responsibility for submitted work.
• Feedback uptake and revision: using AI-supported feedback critically to improve work rather than accepting it uncritically.

The authors reported satisfactory to strong internal consistency and evidence supporting the scale’s validity, including measurement invariance across gender, discipline, and AI-use-frequency groups in their samples. Scores were positively associated with feedback engagement and intention to maintain academic integrity.

These results are promising, but the distinction between association and causation is essential. The study used a cross-sectional scale-development design; it does not show that raising a student’s score will necessarily improve academic integrity or learning outcomes. The instrument is also focused on higher education and needs further testing across languages, cultures, institutions, and observable behaviours. Because it uses self-report items, it complements rather than replaces performance-based assessment.

What should schools actually measure?

Taken together, the research suggests that AI literacy assessment should cover more than prompt-writing. A practical framework for classroom use can focus on five observable capabilities.

1. Understand the tool and its limits

Can the student explain, in appropriate terms, that generative AI may produce plausible but inaccurate answers? Can they identify situations where a model’s output needs independent checking? The goal is not to require every learner to understand the technical details of a large language model, but to develop an accurate mental model of its strengths and limitations.

2. Evaluate evidence, not just fluency

Give students an AI-generated response containing a mix of accurate statements, unsupported claims, and a questionable reference. Ask them to verify selected claims using credible sources and explain which evidence changed their judgment. A fluent answer should not receive full credit simply because it sounds authoritative.

3. Make appropriate decisions about delegation

Students should be able to distinguish between using AI to brainstorm, request feedback, or practise a skill and handing over the central intellectual work that an assignment is intended to assess. The boundary depends on the learning objective and the teacher’s rules. Assessment instructions should therefore specify permitted, restricted, and prohibited uses instead of relying on vague expectations.

4. Disclose use and retain responsibility

Where disclosure is required, students should be able to describe how AI contributed to their work. They should also be able to defend the reasoning, sources, and final decisions in their submission. Disclosure alone does not prove learning, but it can make the process more transparent and support fairer assessment.

5. Use feedback to improve, not merely to finish

Ask learners to compare AI feedback with a rubric or teacher feedback, decide which suggestions to accept, and explain why they rejected or adapted others. This makes evaluative judgment visible and reinforces the idea that AI-generated feedback is a resource to assess—not an instruction that must automatically be followed.

A practical assessment design for teachers

A balanced approach can combine three kinds of evidence rather than relying on one score:

• Knowledge check: a short quiz on AI capabilities, limitations, privacy, bias, and responsible use.
• Authentic performance task: students critique an AI output, verify evidence, improve the answer, and explain their decisions.
• Process reflection: students record what they delegated, what they checked, what they changed, and what they learned.

A simple rubric might assess the accuracy of verification, the quality of reasoning, the appropriateness of AI use, transparency about assistance, and the student’s ability to explain the final work. The rubric should be shared before the task and aligned with the actual learning objective. It should not reward elaborate prompting if prompting is not the skill the course intends to teach.

This approach also helps avoid a false choice between banning AI and allowing unrestricted use. Different assignments serve different purposes. If the objective is unaided recall or foundational practice, AI may need to be restricted. If the objective is evaluating sources, revising arguments, or using tools responsibly, AI can be included explicitly in the assessment design.

Equity and validity: the cautions that matter

Assessment design must account for unequal access to paid tools, differences in language proficiency, disability accommodations, and students’ prior experience with technology. A test that inadvertently measures subscription access, English fluency, or familiarity with one interface may misrepresent the intended competence.

Tools also change quickly. A test tied to a particular product interface may become obsolete after an update. Durable assessment should focus on transferable judgment: checking evidence, recognizing uncertainty, selecting appropriate support, protecting private information, and explaining decisions.

Finally, no single score should be treated as a complete measure of a student’s AI literacy. Self-report questionnaires, knowledge tests, performance tasks, process evidence, and teacher observation each capture different aspects. Their combination is more informative than any one instrument used in isolation.

What educators and institutions should do next

For teachers, the immediate step is to define the capability a task is meant to assess, then design a short activity that makes that capability observable. For institutions, the priority is to provide consistent rules for acceptable AI use, teach students how to verify outputs, and evaluate assessment methods for fairness and validity.

For researchers, the agenda includes testing instruments across age groups and languages, comparing self-reports with actual performance, and studying whether assessment results predict learning over time. New scales should be evaluated not only for reliability but also for whether they measure the intended construct and work fairly across different student populations.

UNESCO’s guidance on generative AI in education and research provides a broader human-centred framework, emphasizing privacy, age-appropriate use, validation, and the development of educators’ and learners’ competencies. These principles support an approach in which assessment is designed around learning and responsibility rather than around the presence or absence of a particular tool.

Key takeaways

• AI literacy is broader than prompt-writing or familiarity with a chatbot; it includes evaluation, verification, appropriate use, ethics, and responsibility.
• Self-report questionnaires can reveal students’ perceptions, but they do not by themselves demonstrate practical competence.
• The GLAT study, involving 355 higher education students, found that a performance-based test predicted GenAI-supported task performance better than self-ratings in that study.
• A September 2026 study developed an 18-item scale focused on AI literacy in assessment contexts, covering criteria, appropriate use, verification, attribution, and feedback-based revision.
• Both instruments have limits and need further validation across contexts; neither should be treated as a universal measure of every AI-related capability.
• Teachers can begin with authentic tasks that require students to verify, justify, revise, and explain their use of AI.

Editorial conclusion: The central challenge for education is no longer only teaching students how to use AI. It is establishing credible evidence that they can use it thoughtfully, verify what it produces, and remain responsible for their learning. Assessment should make those capabilities visible. Confidence and polished output may be useful signals, but neither is a substitute for demonstrated judgment.

Sources and further reading

• Jin, Y., Martinez-Maldonado, R., Gašević, D., & Yan, L. (2025). GLAT: The generative AI literacy assessment test. Computers and Education: Artificial Intelligence, 9, 100436. https://doi.org/10.1016/j.caeai.2025.100436
• Nie, J., Zhang, Z., Lu, X., Zhang, Y., & Zhang, M. (2026). Development and validation of the generative AI assessment literacy scale for higher education students: psychometric evidence and associations with feedback engagement and academic integrity. Frontiers in Education, 11, 1934632. https://doi.org/10.3389/feduc.2026.1934632
• How AI literacy correlates with affective, behavioral, cognitive and contextual variables: A systematic review. (2025). Computers and Education: Artificial Intelligence, 9, 100493. https://doi.org/10.1016/j.caeai.2025.100493
• Lintner, T. (2024). A systematic review of AI literacy scales. npj Science of Learning. https://doi.org/10.1038/s41539-024-00264-4
• UNESCO. (2023). Guidance for generative AI in education and research. UNESCO guidance.

Editorial note: This article synthesizes published research available as of 10 October 2026. Findings from higher education samples should not be assumed to generalize automatically to school-age learners or every educational setting.