Individualized Spaced Repetition in Hierarchical Knowledge Structures

Anki contributor here:

I’ve been using SRS on and off since 2006 and this post addresses one of the key failure modes I’ve seen in SRS usage—they tend to be fantastic for discrete pieces of information, like the names of capital cities, the pronunciation of various symbols or various physical constants.

Most my usage of SRS in the early days was languages. For that domain, SRS is useful for learning an alphabet (or syllabary), for scaffolding enough common words to start understanding graded readers and for a few other tasks. They’re terrible as a primary learning strategy, though! Too much of language is highly nuanced and context dependent and the only way to absorb all the unwritten rules, the common collocations and the precise boundaries of word meanings is through extensive input. This means encountering the same words and structures in countless different variations and contexts, not the same few sentences drilled repeatedly (at least past the beginning stages).

By paying attention to the specifics of learning math and tailoring the SRS to match it, it’s possible to make a far more efficient system than anyone could with an Anki deck. I’d love to see similar efforts in other subjects!

More than a decade ago I worked as a research programmer on a similar AI tutoring product for hierarchical skills like mathematics at Carnegie Mellon (in conjunction with the Pittsburgh Science of Learning Center and Carnegie Learning).

The system would prompt students with problems that incorporated dozens of sub-skills, each of which incorporated other sub-skills and so on. If the student gets the problem right, all requisite sub-skills are marked as reasonably strong. If they get it wrong then the statistical model concludes that at least one sub-skill must be weak/faulty/missing, and when selecting subsequent problems the system would incorporate problems with different subsets of the original sub-skills to identify the missing sub-skill(s), with the goal being to spend time reinforcing only those skills that the student has not mastered.

I no longer work in that domain, but I recall it being amazingly efficient -- it often only took a dozen questions to assess a student's understanding of thousands of skills.

There lots of knowledge types where Anki might not be the ideal choice. For example,

- itemized lists of information. E.g. The 5 reasons for the civil war, type of questions. What to do, if you miss one item? Your card as a whole gets over exposed, while the missed item might get underexposed.

- Related information, e.g., history dates. When using Anki the students tends to latch on to incidental facts, e.g., Treaty of Paris, ah I remember it has two repeating digits, yes, 1899. While it would be more useful for a student to think, ah, that was after the Cuban independence struggles in the 1890's.

Any user will recognize that this effect is very strong. You tend to remember cards by the most trivial of things.

- Hierarchical knowledge. Things like chess openings. How do you put that in Anki cards. It's all a kludge. Where to put the variants, etc.

- Knowledge networks. Things like medical information (where Anki is hugely popular). But typically, you get cards with massive amounts of information because you have lots of linked information, (symptoms, causes, treatment, pharmaceuticals, etc. )

- Even in language learning. We have the useful fiction that there is a 1-1 relationship between words in two languages, but there almost never is.

- Learned knowledge tends to be linked to Anki. It happens often that you can remember things when doing Anki, while being at a loss when needing the information in the real world.

Add to that, that Anki does nothing for conceptual understanding. You really need to learn a subject before memorizing it, but in the learning phase Anki is not helpful.

So in short, yes, Anki is the best tool there is to help learning, but I'm sure better tools can be made, especially when targeted to a particular knowledge field.

Say you have concepts/items/cards A, B and C, with

A -> B -> C (C encompasses B, B encompasses A, keeping the notation from the article).

As I understand it, the article advocates for showing C first, then you can assume that you also know B and A to at least some part, and save yourself the repetitions for these.

Intuitively, I would have guessed the opposite approach to be the best: Show A first, suspend B until A is learned (by some measure), then show B, etc.

That means no repetitions to skip, but also you get less failures (and thus additional repetitions) that occur as follows: you are shown C, but don't know B anymore, and thus cannot answer and have to repeat C.

If you are shown C before B, you kinda make C less atomic (you might have to actively recall both, B and C to answer it), showing B before C makes C more atomic, as you will have B more mentally present/internalized and can focus on what C adds to B.

I've been wanting to request an Anki feature for a while that is in this vein (and would enable this style): allowing cards within a deck to be tagged with a trigger, which would activate when all (or some portion of) cards within a deck also tagged with that trigger reach "maturity." When activated, it can add a tagged batch of cards from the deck as new cards for review, or more generally it could change their state (from/to Young, Mature, Suspended, etc.).

This would allow cards to be automatically introduced in groups, rather than the "just give me 10 new cards every day" thing. It could also allow a default suspended deck style, where all but a few cards begin suspended, and they are unsuspended in an orderly way based on one's progress with the active cards. That could be a pattern that works for drilling hierarchies of knowledge.

There's a deck style called the KOFI (conjugation first) system for learning Romance conjugations that would really benefit from that, which now depends on people manipulating the decks manually with the aid of an abused deck description field describing that manipulation.

Anki is pretty hostile to order, because spaced repetition is all about moving individual cards optimally and without regard to other cards, but the order in which information is introduced can be important. At this point you can only choose between introducing cards in a linear order, or completely at random. There's no way to deviate from that without the user having to learn to juggle things manually. A deck creator can't just suggest a schedule programmatically, even if the deck was designed around that schedule.

The other half of this, where cards that duplicate combinations of other cards are automatically strengthened when their related cards are strengthened (or vice versa) is interesting. Seems like a neuron model.

I really hope to see more research like this. I feel like LLMs have insanely high pedagogic value, and that's because I've been using them to teach myself difficult subjects and review my understanding.

The issue with the kind of system that the author is proposing for curious self-driven learning is that the SRS is optimized for a given curriculum.

Many people including myself use flashcards to guide (and sort of "create") their own curriculum. Flashcards with SRS are really good for many things, but it's difficult to generalize them for thus usecase.

I'd really like to see some prototypes of people integrating LLM intelligence in creating and adjusting cards on the fly. It's clearly something LLMs excel at, especially the larger closed-source ones like Claude 3.5 sonnet

This is why my initial flashcards on a topic would be cover a breadth of individual details, but later I'd start creating concept based flashcards where I'd ask things like "how does it make sense that....?"

After somewhat memorizing foundational knowledge, understanding concepts and having frameworks of thinking for a topic will further increase critical thinking while reducing the amount of cognitive effort required for critical thinking about unfamiliar situations.

I hope one day these more efficient models that "clean up" the review schedule by linking/identifying related flashcards become publicly available.

Now these are the insights I'm looking for when it comes to "second generation" SRS software suites. When you focus your attention on a specific niche, like mathematics here, you can get some serious improvements in retention just by using the shape of the subject to your advantage like this.

Hi Jason, we had discussions at Caltech on some of these ideas many years back. We do have a patent, now lapsed, on doing spaced repetition on knowledge graphs: https://patentimages.storage.googleapis.com/ff/ae/6d/3483859...

Math pedagogy in the US leans to understanding being more important than practice. This is wrong. As you point out, understanding more often comes from practice. To correct this imbalance, software like your company is developing is useful, as are platforms like ALEKS, iReady, etc.

The challenge is addressing the social aspect of learning -- the benefits students get from working and learning together. Is it still a classroom if all the students are doing different things?

Of course, students come in with different abilities and levels of preparation. Differentiation that works in a class setting is very hard. Software is the easier part. Students who are motivated to learn and can grow with guidance from a computerized tutoring environment -- helping them is also the easier part.

While this article presents a system it doesn't present any results. Does this modification to SRS help? Which type of student does it help? How large is the effect if there is one?

I haven't read the larger pdf of which this is a part so perhaps someone who has can provide a pointer to some results.

Our company (Language Zen) has this patent https://patents.google.com/patent/EP1891619A2/en and we have done work to teach hierarchical knowledge for language learning

Is anyone building a personal social friend graph / calendar hangout maximizer with spaced repetition?