Educational recommendation systems: a parent guide

A note from me this week

This week's issue is about educational recommendation systems. But first, my week.

I've been taking it easy this week, so as to relax myself and swap into a marathon mode of sorts. A few adjustments to the newsletter, and we even got our first subscriber.

The device is at a standstill. Yet I've watched a few videos of people able to achieve similar devices via the ESP32, a small, low-cost microcontroller chip.

I learned this week to celebrate the small wins and focus on what matters in the business. Validation is way more important than product perfection.

I started Archimedes for a few reasons. Education has always been important to me, especially seeing how money could influence the quality of education. As someone in public school, I always thought of this as unfair yet unchangeable. Now, with AI, the time is right and I can actually level the playing field.

Education also means more intelligent people, and they'll be inclined to create a better society. That's part of my mission to give back to people who have helped me. I also think that the current system is broken and doesn't include or adapt to everyone.

I'm starting to notice the extreme polarity in regard to education. Many believe the one-teacher-to-many-students method to be failing, and are unsure if technology is the answer. It makes me realise just how much research and guesswork we still have to do at Archimedes. Still, I'm bullish on education, as many are starting to question the schooling system and its applicability in the modern era.

I've also recently discovered recommendation systems, or recsys. There are a few scholarly articles online on educational recommendation systems, and that's what the rest of this issue is about.

What is an educational recommendation system?

Your child finishes a maths question in an app, and a new one appears. Who picked it, and why that one?

That's a recommendation system at work, often shortened to "recsys". It's software that sifts through lots of options and predicts which one suits a particular person. You already use them. This Databricks guide, written by a former YouTube recommendations lead, explains how they narrow a billion possible YouTube videos down to a few hundred for you.

An educational recommendation system does the same job with learning. Instead of the next show, it suggests the next exercise, video, topic or course.

Netflix explains that its system looks at your viewing history, members with similar tastes, and details about each title. Learning apps borrow those ideas and add one more:

• Collaborative filtering: "students like yours did well with this next."
• Content-based filtering: "you worked on fractions, so here's more fractions."
• Knowledge-based: it follows rules about the subject, such as which skills come before others.

Most real systems mix these together, which researchers call a hybrid approach.

How is it different from Netflix or YouTube?

Here's the big difference. Netflix wants you to keep watching. As the Databricks guide notes, recommender models typically predict things like clicks, purchases and watch time.

Learning is different. The goal isn't more time in the app. It's for your child to be able to do something they couldn't do last week.

That changes what a "good" recommendation looks like. Sometimes the best next step is a harder question. Sometimes it's stepping back to an earlier idea that's still shaky. A system chasing clicks would rarely choose either.

To make good choices, an educational system needs a sense of what your child has actually mastered. One well-known method is Bayesian Knowledge Tracing, which turns each answer into an updated estimate of whether a skill is secure. We walked through the maths in Bayesian Knowledge Tracing, explained.

What does the research say about educational recommendation systems?

The honest answer: the idea is promising, but the evidence is thinner than the marketing.

Most studies aren't about school-age children. A 2021 systematic review looked at 98 studies and found most systems were built for formal education, mainly universities.

Many systems are judged on the wrong thing. A 2024 review of 28 papers found fewer than half tried to optimise learning, and only about a third measured learning outcomes at all.

Fun isn't the same as learning. In one classroom study, 190 students in grades 7 and 8 used either a popular equation-solving game or a research-based tutoring system for five class periods. The game group solved more problems and enjoyed it more. The tutoring group did significantly better on the test afterwards. The researchers' conclusion is worth remembering: "intuitions about what works, educationally, can be fallible."

Step-by-step tutoring can work well. A major review by Kurt VanLehn found intelligent tutoring systems were "nearly as effective as human tutoring." What happens after the recommendation matters most.

What can go wrong?

A few problems come up again and again in the research:

• Cold start. A new student has no history, so early recommendations are guesses.
• Over-specialisation. A system can keep serving more of the same, a bit like a filter bubble. Google's course on recommendation systems describes a final "re-ranking" step partly to keep results diverse and fair.
• Bias. In a review of algorithmic bias in education, Ryan Baker and Aaron Hawn found that models trained on one group of students often work worse for others. Most of that research comes from the United States, so we know even less about how these systems treat Australian students.

None of this means avoiding learning apps. It means asking a few questions.

What should parents ask a learning app that recommends content?

You don't need to understand the maths. These five questions will tell you a lot:

1. What is it trying to maximise? Learning, or time spent in the app?
2. How does it decide my child has "got" a skill? Look for evidence from answers over time, not minutes logged.
3. Can it step back as well as forward? Good systems revisit shaky foundations.
4. Can I see why it chose something? You should be able to tell what's secure and what isn't.
5. What data does it keep, and where?

These line up with the Australian Framework for Generative AI in Schools, which expects vendors to help users understand how their tools work and their potential biases, and schools to be told what student data is collected.

If minutes are all an app reports, our piece on screen time and learning explains why that's not enough.

You can also ask your child. Tonight, try: "Why do you think the app gave you that question next? Was it too easy, too hard, or just right?" If they're stuck, our guide to helping kids with homework without doing it for them has more scripts.

> Key takeaway: An educational recommendation system should be judged by what your child can now do, not by how long they stay or how much they enjoy it. Ask what an app optimises for, how it decides a skill is mastered, and whether it can show you why.

Where does Archimedes fit?

Archimedes is a patient AI tutor for kids, being mapped to the Western Australian Curriculum. It has a recommendation layer of its own, and it's built on Bayesian Knowledge Tracing.

Every skill on our skill map carries its own mastery estimate, and every answer nudges it. That estimate helps choose the next question: build on what's secure, or step back to an earlier idea that's still shaky. The parent view shows you what's secure, what's shaky and where your child is stuck, skill by skill.

There's also the Archimedes device, our hardware side, which as I said above is at a standstill for now.

We don't have all the answers. As the research shows, nobody does yet. But we'd rather build toward learning than toward screen time.

FAQ

What is an educational recommendation system?

An educational recommendation system is software that suggests what a student should learn or practise next. It might suggest an exercise, a video, a topic or a course. It works like Netflix or YouTube suggestions, but the goal should be learning rather than keeping someone watching for longer.

Do educational recommendation systems actually work?

They can, but the evidence is still limited. Systematic reviews find that most studies focus on university students, and many never measure whether students actually learned more. Fewer than half of the systems in one 2024 review tried to optimise learning, so it's worth asking any app how it measures progress.

How is a learning app different from YouTube recommendations?

YouTube's suggestions are designed to keep you watching. A good learning app should do something harder: pick the next step that helps your child learn, even when that means a tougher question or going back to basics. Engagement and enjoyment matter, but research shows they aren't the same as learning.

---

If this was useful, subscribe to the Archimedes newsletter for a plain-English issue on learning and AI each week.