My Tutor Hour

Insights on teaching, learning and the science behind them

Abstract illustration of research on generative AI in higher education: a streetlamp lights a few study cards while many others sit in the dark

Lots of Studies, Fewer Answers: What the Evidence Really Says About AI in the Classroom

A flood of studies

If you’ve tried to keep up with research on generative AI in higher education over the last few years, you’ll know the feeling. A new study lands in your inbox every other day. One says chatbots boost achievement. Another says they make students lazy. A third finds that students like them very much, which you could probably have guessed.

I’ve written about several of these studies on this blog, and I’ve enjoyed doing it. But I’ve started to wonder whether we’re seeing the wood or just a great many trees. So I was glad to come across a new paper in the International Journal of Educational Technology in Higher Education that steps back and looks at the whole forest.

In short, a team mapped 394 studies of generative AI in higher education, published between late 2022 and the end of 2025. They found that most of the research measures what’s easy to measure, over short periods, and that some of the things we care about most have hardly been studied at all.

How the researchers mapped generative AI in higher education

The review comes from a team based in Vancouver: Michael Pin-Chuan Lin, Daniel H. Chang, Philip H. Winne, G. Michael Bowen and Jeeho Ryoo. If Winne’s name rings a bell, it’s probably because he’s spent decades researching self-regulated learning, how students plan, monitor and steer their own studying. Keep that in mind; it comes up again shortly.

The team searched for empirical studies of generative AI in higher education teaching and learning, starting from ChatGPT’s public release in November 2022 and running to the end of 2025. They ended up with 394. For each one, they recorded how the AI was used, what the researchers were trying to measure, and how they measured it.

It’s worth being clear about what kind of study this is. A scoping review is a map, not a verdict. It doesn’t pool the results to tell you whether AI improves learning by some precise amount. Instead, it shows you where researchers have been looking, what they’ve been counting, and which bits of the territory nobody has explored yet. For a field moving this fast, that’s surprisingly useful.

Generative AI in higher education: what gets studied, and what gets missed

The first finding is a quietly revealing one. In most studies, AI wasn’t transforming how teaching works. It was slotted into things universities already do, especially writing, tutoring and giving feedback. The authors describe most adoptions as pragmatic, which strikes me as the polite academic way of saying “people used it to get existing jobs done faster”.

Then there’s what researchers chose to measure. The most common outcomes were engagement, usability, academic performance, writing quality and critical thinking. All reasonable things to look at. But several areas that matter a great deal got far less attention:

  • Self-regulated learning: whether students are still learning to plan, monitor and manage their own studying, or handing that over to the chatbot.

  • Ethics: how students and staff navigate questions of honesty, authorship and fairness.

  • Creativity: whether AI widens or narrows the range of ideas students produce.

  • Feedback quality: not just whether students received AI feedback, but whether it was any good, and whether they could tell.

  • Educational equity: who benefits, and who gets left behind.

The methods tell a similar story. Many studies relied on short-term surveys, tests or ratings of finished work. That makes sense when you’re trying to publish quickly in a fast-moving field, but it means much of the evidence is about how students felt or how well a single task turned out, not what they could do weeks or months later.

The review isn’t alone in spotting this. Digital Promise, a US non-profit, built a collection of more than 300 studies on AI in higher education and found a similar heavy focus on writing and assessment, along with gaps such as data privacy and patchy reporting of how the AI tools were actually set up.

Finishing the task isn’t the same as learning

Why does it matter that so much research is short-term and focused on finished work? Because with AI, getting the work done and learning from it can come apart in a way they rarely did before.

Two well-known studies from outside this review show the problem nicely.

In one, researchers working with nearly a thousand secondary school maths students in Turkey gave some of them access to a standard chatbot during practice sessions. Their practice scores shot up. But when the chatbot was taken away for the exam, those students did worse than classmates who’d never had it. A second group used a version designed to act like a tutor, giving hints rather than answers. They also did much better in practice, and they didn’t suffer that drop in the exam. Same technology, very different results, depending entirely on how it was set up.

In the other, a team at Harvard compared an AI tutor, carefully designed around good teaching principles, with an active-learning physics class. Students using the AI tutor learned more, in less time. That’s a genuinely encouraging result, but again, it came from a tool built deliberately to make students think, not one that simply hands over answers.

Put those side by side and a pattern emerges. If you only measure how well students complete a task while the AI is helping, you can’t tell the first kind of tool from the second. Both look great. You only see the difference when you take the AI away, or wait a while, and check what’s actually stuck. That’s exactly the kind of measurement the review found in short supply.

It also echoes the study I wrote about in Working With AI: the student teachers who got the most out of AI weren’t the ones who leaned on it hardest, but the ones who kept judging and revising what it produced.

Four questions worth asking

The number of AI studies is growing fast, but evidence that a tool is convenient or popular isn’t the same as evidence that it produces lasting learning. Whether you teach in a university, a school or across a tutoring table, these four questions seem to me the right ones to bring to any new AI tool or any new study about one.

Does it improve understanding, or just help students finish? A polished essay or a completed worksheet tells you the task got done. It doesn’t tell you who did the thinking.

Can learners still do it without the AI? This is the simplest test, and the one most often skipped. A quick task with the tool switched off, a week or two later, tells you more than any satisfaction survey.

Are students learning to judge the feedback they get? AI feedback is fast and fluent, but not always right. Being able to tell good feedback from bad is a skill in itself, and one we’ll need students to have for the rest of their lives.

Who benefits most, and who might be left behind? Confident, well-supported students may use these tools brilliantly. Others may use them to avoid the very struggle that would have helped them. Without looking, we won’t know which is which.

Looking under the lamppost

There’s an old joke about a man searching for his keys under a streetlight. A passer-by asks where he dropped them. “Over there in the park,” he says. “But the light’s better here.”

I thought of that joke more than once reading about this review. It’s not a criticism of the hundreds of researchers involved, most of whom were doing careful work under real pressure to say something useful quickly. It’s just how research tends to go. We study what’s easy to see. Engagement scores, usability ratings and the grade on a finished essay sit right under the lamppost. Self-regulation, creativity and the slow build-up of understanding over months are out in the dark of the park.

There’s even a name for the trap at its worst. It’s sometimes called the McNamara fallacy, after the US defence secretary who leaned heavily on what could be counted during the Vietnam War: measure what’s easy, then decide that what you can’t easily measure doesn’t really matter, and finally that it doesn’t exist.

I don’t think education has fallen that far with AI. But it’s a useful warning. The things that are hardest to measure in learning are often the things we care about most. Whether a student can sit with a hard problem. Whether they know when they don’t understand something. Whether they’d recognise a bad answer if a confident machine handed them one.

It seems fitting that one of the authors has spent his career on self-regulated learning, perhaps the hardest of all these things to see from the outside. The review is, in a way, a polite request to bring a torch into the park.

A note of caution

This is a scoping review, so its job is to map the research, not to give a single answer on whether AI helps or harms learning. It won’t tell you to adopt or ban anything.

It also has its own boundaries. The team searched English-language, peer-reviewed journal articles in two databases, so studies published in other languages, in conference papers or in less mainstream journals won’t be included. The evidence window closed at the end of 2025, and AI tools have kept changing since then. Some of the gaps it identifies may already be starting to close.

Still, I found it one of the more useful things I’ve read on the subject this year, precisely because it doesn’t add another headline about AI being wonderful or terrible. It tells us where we’ve been looking and where we haven’t. For anyone trying to make sensible decisions in a classroom or a lecture hall, that’s a good place to start.

The next time you see a study claiming an AI tool “improved learning”, it’s worth asking a couple of quiet questions. Learning of what, exactly? Measured how, and when? And would the students still know it if you took the tool away?

Sources

Leave a Reply

Your email address will not be published. Required fields are marked *