When an AI Study Assistant Backfires: A 2,379-Student Trial

University student studying at a desk near large library windows, illustrating research on AI study assistants and student engagement.

Research · Education · Practice

Generative AI tools are increasingly being integrated into university learning platforms on the assumption that on-demand explanations and answers will make study more efficient. A randomized university trial at the University of Maryland complicates that assumption. In the Fall 2025 semester, researchers evaluated access to a course-integrated virtual study assistant across multiple disciplines. The working paper reports lower final grades and reduced engagement with the university learning-management system among students offered access in the principal within-course comparison.

The findings are important, but they require careful interpretation. The study evaluated one institutional tool, under particular conditions of implementation and uptake. It does not establish that AI tutoring in general harms learning, nor does it identify a single mechanism that explains the observed differences.

The study and its design

Liu and colleagues examined the University of Maryland’s Virtual Study Assistant (VSA), a generative-AI tool embedded in the university’s learning-management system. It was built on GPT-4o and used retrieval-augmented generation to draw primarily on course materials selected by instructors. During Fall 2025, the study involved 2,379 undergraduate students and 30 instructors across multiple disciplines.

The researchers used a randomized controlled design to evaluate access to the tool. The working paper reports a particularly informative comparison between sections of the same course: students offered access to the VSA had final grades lower by 0.37 standard deviations, while recorded participation in the learning-management system was lower by 0.90 standard deviations. The authors also report that estimated academic losses were larger for first-generation students. These are standardized effect sizes, not percentage-point changes; the university’s summary describes the grade difference in the matched-course comparison as approximately four percentage points.

Source: Liu, J., Sweet, T., Chen, M. H., Engelberg, J., Clark, M., Persaud, A., Lancaster, A., Hollingsworth, J., Rice, J. K., & Masters, M. (2026). The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial. EdWorkingPaper No. 26-1598, Annenberg Institute at Brown University. https://doi.org/10.26300/y3f8-vh05. This is a working paper and should not be described as a peer-reviewed journal article.

Access is not the same as use

One of the study’s most consequential details is the low uptake: only about 15% of students offered the VSA used it at least once. The reported estimates therefore primarily concern the consequences of being offered access in this institutional setting—not the effect of intensive use by a typical student, and not a clean comparison between frequent AI users and non-users.

Among students who did use the assistant, reported query patterns also raise questions about how the tool functioned in practice. Approximately 73.8% of requests sought information, explanations, or solutions; 11.3% concerned practice or test preparation; and 0.7% sought feedback on the student’s own work. The tool had a default mode that could provide direct answers, while a tutoring mode was designed to guide students toward a response. Few instructors switched away from the default setting.

These observations are consistent with a possible mismatch between the intended role of an AI “tutor” and the role students actually assigned it: quick information retrieval or answer delivery rather than sustained practice. However, usage patterns alone do not prove that answer-seeking caused lower grades. They are descriptive evidence that helps frame hypotheses for subsequent research.

What might explain the results?

Several mechanisms are plausible, and the available evidence does not establish which one is decisive.

Displacement of course engagement. Students may have relied on the assistant instead of opening readings, revisiting lecture materials, or completing other activities in the learning platform. The observed decline in recorded platform participation is consistent with this possibility, but LMS activity is an imperfect proxy for learning: fewer clicks do not necessarily mean less understanding.
Answer substitution rather than guided practice. When a system supplies a solution before a learner has attempted the task, it may reduce opportunities for retrieval, explanation, error correction, and productive struggle. The request patterns make this hypothesis credible, but the trial was not designed to isolate the cognitive mechanism.
Implementation and trust. The tool was integrated into a course platform where instructors could access chat logs. Awareness of that context, alongside access to other tools such as ChatGPT and Gemini, may have influenced who used the VSA and what they were willing to ask. This limits comparisons between the institutional assistant and private, standalone AI tools.
Instructional configuration. A tool’s effects may depend on whether it provides direct answers, prompts students to explain their reasoning, connects responses to assigned readings, or is integrated into assessment and classroom routines. The results should not be treated as a test of every possible tutoring design.

Important limitations and unresolved questions

First, the paper is a working paper, not yet a peer-reviewed publication. Its estimates should be treated as provisional and revisited if the authors release revised analyses or a journal version.

Second, the intervention was access to a specific course-integrated assistant at one large U.S. public university. Its findings cannot automatically be generalized to other age groups, countries, subjects, models, or AI systems designed around structured tutoring. The results also do not show that AI use outside the VSA produced the same effects.

Third, low uptake matters for interpretation. An estimate of the effect of being offered access answers a practical policy question—what happened when this university made the tool available under these conditions?—but it does not tell us what would happen if students were trained to use a different tool intensively and appropriately.

Fourth, reduced learning-management-system activity is a behavioral measure, not a direct measure of conceptual understanding, retention, transfer, or long-term learning. Lower activity may signal displacement of course work, but alternative explanations should be tested. The larger estimated losses among first-generation students also warrant careful follow-up; they should not be interpreted as proof that AI tools inherently widen educational inequality.

Finally, randomized field trials estimate the effect of an intervention package in a real setting. They do not necessarily identify the causal pathway behind an outcome. Access, faculty choices, tool configuration, students’ existing AI use, and the surrounding course environment may interact. The trial is therefore a reason to investigate deployment conditions, not a universal verdict on AI tutoring.

Implications for educators and institutions

The most defensible practical conclusion is that access to an AI tool should not be assumed to constitute an effective learning intervention. Institutions should define the learning purpose before adoption, explain how student data and conversations are handled, and evaluate the effects on actual learning rather than satisfaction or convenience alone.

Prefer scaffolding over immediate completion. Where appropriate, configure assistants to offer hints, ask learners to show an initial attempt, diagnose errors, and reveal worked solutions in stages.
Reconnect AI use to primary materials. Ask students to identify the relevant reading, lecture segment, evidence, or worked example after using the assistant.
Make learning visible. Use low-stakes retrieval practice, short explanations in students’ own words, in-class application, and assessments that test transfer—not just the answer produced during an AI-supported task.
Monitor outcomes and equity. Compare engagement, performance, and student experience across groups, while avoiding surveillance practices that are disproportionate or poorly explained. Be prepared to revise or pause a deployment when it displaces useful learning activity without demonstrable benefit.
Evaluate alternatives experimentally. Future studies should compare direct-answer systems with guided tutoring, measure learning after AI support is removed, track longer-term retention and transfer, and report implementation fidelity and subgroup uncertainty.

These are design recommendations derived from the study’s findings and wider learning-science principles; they were not all directly tested in this trial. Their effectiveness should itself be evaluated rather than assumed.

How this fits the wider evidence

The Maryland result should be read alongside, not instead of, other research. Bastani and colleagues’ randomized study in a high-school mathematics setting found that access to a general-purpose GPT interface could improve performance while it was available, yet some gains did not carry over to an unaided assessment; a more constrained tutor designed with learning safeguards produced a different pattern. That study is not a direct replication of the Maryland trial: the learners, subject, intervention, and outcomes differ. It nevertheless supports a useful research question—whether an AI system helps students learn independently, rather than merely complete the immediate task.

Reference: Bastani, H., et al. (2025). “Generative AI Can Harm Learning.” Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2422633122.

For the broader institutional context, see the OECD’s Digital Education Outlook 2026, which discusses the need to evaluate generative AI in relation to educational goals, teacher agency, and learning processes rather than tool availability alone: https://doi.org/10.1787/062a7394-en.

Conclusion

The Maryland trial provides a cautionary empirical result: in one large university implementation, offering access to a course-integrated AI study assistant coincided with lower final grades and substantially lower recorded engagement with course materials. The result is important precisely because it resists a simple technological narrative. Yet the paper does not establish that AI tutors as a class are harmful. It shows that design, uptake, instructional context, and the learning activities a tool may displace need to be measured—not assumed.

For universities, the relevant question is not simply whether students can access AI, but under what conditions its use supports durable understanding, independent performance, and equitable learning outcomes.

Image credit: Austin / Unsplash.