Category: Articles

Articles on learning, spaced repetition, productivity, data science, programming, and more.

What are your “Big Projects”?

One thing that has become clear to me as I get older and raise a family is that human beings need a bigger purpose in life. We need big goals and big dreams that we can endlessly pursue. Without this, we inevitably stagnate and decline.

You can clearly see our need for a higher purpose throughout society. You see this in the great religious traditions that provide followers with meaning, structure, and direction. You see this in the advice of successful and influential people such as Jordan Peterson that encourage people to “aim high”. You see this in the success of industry titans like Steve Jobs or Elon Musk, who propelled their workers and themselves to great heights using a big vision and persistence in reiterating it at every step of the journey.

Clearly, there are many ways of working toward big goals and there are many goals you can work toward. I want to focus on one technique that I’ll call “The Big Project”.

Characteristics of the Big Project

The Big Project is a significant project that provides you with a platform to learn many new things. A good Big Project should have a few defining characteristics.

  • It is Ambitious. The project will be ideally working towards a broad, lofty, and exciting goal. As Peter Drucker says in his book “The Effective Executive“: “Aim high, aim for something that will make a difference, rather than for something that is “safe” and easy to do.” (pg. 111)
  • It Matters to You. The goals of The Big Project should be aligned with your personal values and beliefs, and propel you toward an outcome you care about. If you don’t care about the Big Project, you won’t stick with it. Similarly, you want The Big Project to be building skills that are actually relevant to you.
  • It Matters to Others: The Big Project should have some tangible real-world relevance to others, rather than just being a “toy” project for you.
  • It has Identifiable Milestones and Subtasks: Although the project should be large in scope with no clear “project complete” criterion, you should still be able to complete well-defined subtasks along the way to help you stay motivated and ensure the project is meeting your goals. These subtasks should be achievable, but also provide room for growth. You want a challenge, but not so much difficulty that you get overwhelmed and demotivated.
  • It is flexible and adaptable: A good Big Project should be flexible and adaptable, and allow for changes and adjustments as needed. This can help you stay on track and achieve your goals even if unexpected challenges or roadblocks arise.

Why Pursue a Big Project?

First and foremost, working on a Big Project is satisfying. It can provide excitement and some sense of meaning if you are thoughtful about what you choose to work on. It can boost your confidence and self-esteem, and provide a sense of accomplishment.

The Big Project takes advantage of the power of compounding, as the philosophy is to consistently build something over a long period of time. Through the magic of compounding, you may reach heights you didn’t expect and couldn’t achieve if you only worked on quick and shallow projects.

The Big Project also provides focus. With a clear overarching goal that inspires you, you will less likely be led astray by activities that don’t propel you toward that vision. You also become less likely to fall into the trap of never-ending planning, because it becomes much more clear what you should work on.

Examples

The Big Project can take many forms, depending on your goals and interests. Some more common Big Projects would include raising a family or living your life in accordance with a particular religious moral code. Both of these things are Big Projects in my life and many others.

In my professional life, one of my main Big Projects is my website, which includes Articles on efficient learning, spaced repetition, productivity, programming, and data science, as well as Download Mark’s Brain, which is a full stack web application that I built for sharing notes and flash cards, synced with my personal collections in Anki and Roam Research.

Together, this Big Project meets all the characteristics required.

First of all, It’s ambitious and large in scope: there is no limit to articles I can write and articles and books I can learn and take notes on. I also have long-term ambitions for Download Mark’s Brain to develop it into a collaborative flashcard development site with the lofty goal of converting all of human knowledge into testable Q+A format.

Furthermore, it matters to me personally for many reasons:

  • Writing is a skill I want to develop and it also clarifies my thinking and learning
  • I want to develop my skills in the myriad of programming patterns and technologies involved in developing a full-stack application
  • I want to grow and maintain my personal knowledge management system
  • I want to maintain a consistent and fast-paced learning / reading cadence
  • I want to eventually build a self-sustaining business

It also matters to anyone else that benefits from my writing and shared notes and flashcards, it is easy to create clear milestones and subtasks, and there is great flexibility in the direction I can take the project.

Conclusion

I challenge you to find at least one Big Project in your life. Whether it’s raising a family, living according to a particular moral code, or working towards a professional goal, a big project can be a powerful tool for your well-being, development, and growth.

Why Take Notes?

Photo by Kelly Sikkema on Unsplash

You can access an Anki deck and Roam Research notes on this article here.

My personal knowledge management system (the “Triple-Pass System”) has two key pillars:

  • Note-taking (AKA “second brain”) to store consolidated information on the content that I read
  • Spaced repetition to retain the most important information from my notes forever, at minimal cost

You might wonder: why is note-taking necessary at all when you have the powerful mental prosthetic of spaced repetition? Spaced repetition makes memory a choice. Why put in the extra effort to take notes when you can just add stuff to your spaced repetition system and be done with it forever?

There are a few reasons why note-taking deserves a place in my personal knowledge management system:

  • Preparing for spaced repetition
  • Retrieval and elaboration practice
  • Computer-aided information retrieval and idea generation

Preparing for Spaced Repetition

It’s usually not a good idea to add information directly to your spaced repetition system while you are reading content for the first time. You are much more likely to add unnecessary information or miss the crux because you lack context from reading the whole piece.

Instead, using your notes as an initial place to store information gives you time to let the information “stew” before you add it to spaced repetition. This helps you faithfully follow rule number 1 of formulating knowledge: do not learn if you do not understand.

Taking notes first has the added benefit of flexibility over when you add material to spaced repetition. You can add to spaced repetition right away, or you can do it later when you have more time. In contrast, going straight to spaced repetition requires you to either add material right away or re-read the entire source document later, essentially starting from scratch.

Retrieval and Elaboration Practice

The research literature on efficient learning tells us that retrieval and elaboration (i.e. recalling things you have learned and re-interpreting them) is extremely beneficial.

This is exactly what you do when note-taking:

  • Review the material (which usually requires some recall)
  • Consolidate it into a form that’s easily consumable (elaboration)
  • Make connections with existing knowledge (recall and elaboration)
  • Intersperse ideas and alternate interpretations throughout your notes (elaboration)

Computer-Aided Information Retrieval and Idea Generation

If you think of your brain as a computer, your spaced repetition system is like your RAM: quickly accessible information. This is ideal for knowledge that you will benefit from being able to retrieve quickly.

In contrast, your note-taking system is like a hard drive: it’s slower to access (since you have to open your note-taking app), but it is capable of storing much larger quantities of information. This is great for looking up details that were not practical to commit to spaced-repetition.

Most digital note-taking tools also have sophisticated search functions that allow you to efficiently look up what you need. These features not only help with information retrieval, but also idea generation by helping you make unexpected connections with other knowledge. Roam Research is particularly good in this area, using a graph-based data model that lets you explore connections between your notes.

How to Improve your Flashcard Knowledge Base

Even the best flashcard developers among us create bad cards on a regular basis (e.g. too long, ambiguous, useless information).

Given the reality that we all are highly imperfect at developing our flashcards, what should we do to improve these crummy cards as they age so you can spend less time reviewing and remember the concepts better?

I call this process flashcard “refactoring” (a term borrowed from software development).

Why refactor flashcards?

Reviewing old flashcards requires time and effort. Here are a few reasons why it’s worth the price:

  • It improves your understanding of the material. The process of breaking learning material down into the smallest “chunks” possible that fit onto flashcards is an extremely valuable exercise. Reviewing troublesome cards clarifies what you don’t understand and forces you to restructure your knowledge in a way that makes sense.
  • Your worst flashcards take up a disproportionate amount of time and effort, while yielding the worst results in terms of retention and usefulness. Following the 80-20 rule, 20% of your cards leads to 80% of the effort in review. So it’s a high value activity to hunt down this subset of your cards.
  • It provides knowledge construction training. Creating good flashcards is a nontrivial skill built over time. You can read Poitr Wozniak 20 rules for formulating knowledge, but actually observing your own performance on your cards and troubleshooting improvements takes your skills to the next level.

The process I use has two broad steps: selection and revision.

Selecting Problem Cards

I use two main methods to find cards needing review.

The first and most important method is finding cards I keep failing (“lapse” is the term in Anki). In the Anki browser, I can use the command prop:lapses>n to find the cards that have lapsed over n times. For me, cards are never lapse more than 8 times because Anki then marks it as a leech and automatically suspends it. Cards that have lapsed 5 or more times are great candidates for refactoring.

The other method is “marking” cards during review when I notice a card is poorly formed. I also try to make notes on marked cards describing what’s causing the problems (coming back to the cards at a later time, you can easily forget the specific issue that tripped you up).

Reviewing and Revising Problem Cards

The first step examining a difficult card is to ask whether I need this knowledge at all. If not, that’s the end of the process – I just delete the card and I’m done with it. I may also revisit the source material or do some Googling on the topic, which will sometimes reveal that the card is pointless or inaccurate.

If I decide that it’s important and relevant knowledge I want to keep, then I’ll examine the card for issues, using the principles from Poitr Wozniak Twenty Rules of Formulating Knowledge.

Example

Consider this data engineering card from my deck which was recently giving me problems:

  • Side 1: Tail latency amplification
  • Side 2: Even if small % backend calls slow, chance of getting a slow call increases if user request requires multiple backend calls, and so a higher proportion of end-user requests end up being slow.

First off, is this card relevant and worthwhile? For me, the answer is definitely yes: it’s both relevant to my job as a data scientist and my software engineering side projects.

Next, diagnose the problem. On closer examination, there are a few things wrong with the card:

In cases where I add material I don’t fully understand, I find the best approach is to go back to the source (which in this case is the book Designing Data-Intensive Applications by Martin Kleppmann). I then refactored the card like this:

  • Side 1: Tail latency amplification (Kleppmann)
  • Side 2: Multiple back-end calls for a single user request increases chance of encountering a tail latency. (Kleppmann)

As you can see, I added a source to clarify where the information came from (Rule 18: Provide source).

I was curious what other cards I have about tail latency, and it turns out there are none! Seems ridiculous to have a card about tail latency amplification, but not have a single one about tail latency which is a more common term. Not having this in my deck probably contributed to interference since I never tested myself on the distinction between the two concepts. So I added:

  • Side 1: Tail latency (Kleppmann)
  • Side 2: High percentile response time. (Kleppmann)

Note that the tail latency amplification card uses tail latency in its response. I’m hoping this will limit confusion between the two and emphasize the distinction (Rule 13: Refer to other memories). I also italicized amplification, to hopefully further avoid interference.

Since I’ve made these changes, I haven’t had any problems with these cards and feel like I have a better grasp on the material. Consider doing the same for the important knowledge in your decks causing you trouble.

The Triple-Pass Method to Remember What you Learn, Forever

You read books. You watch videos. You listen to podcasts. There is a firehose of high-quality information available at your fingertips.

But how much do you actually get out of your media consumption? How much do you remember? For a long time, my answer was “not a whole lot”. I was often shocked at how few key points I could recall from material I had read only days or weeks earlier.

To me, this is unacceptable. Why bother doing so much reading if I’m just going to forget it all? Sure, if reading for entertainment, then no big deal. But most of the material I spend my time on is highly relevant to my professional and personal life. I want this knowledge to compound.

I’ve spent years experimenting, testing, and tweaking systems to get more value out of the media I consume and remember key points forever. I’ve eventually landed on a solution that I want to share with you: the “triple-pass” system. Using this system, I’m confident I’m getting everything I can out of my reading, and not a moment is wasted.

High-level Overview of the Triple-Pass System

At the highest level, the triple-pass system consists of the following stages:

  • First Pass: Active Media Consumption. Take notes and highlight important points from the media you consume (e.g. books, articles, videos).
  • Second Pass: Consolidate and Summarize Notes. Export notes and highlights from the first pass to a central note-taking system, then review, refine, and consolidate. This pass produces what Tiago Forte calls your “second brain“: an external system storing your knowledge in a format that can be easily searched and retrieved for use.
  • Third Pass: Commit to Long-Term Memory using Spaced Repetition. Add the most important parts of your notes from the second pass to a spaced repetition system. This stores the information efficiently in your long-term memory so you can access it in-the-moment when needed.

I go into each of these three stages in more detail below, including the technologies I use at each stage.

First Pass: Active Media Consumption

At this stage, you consume the media that you want to absorb. The key feature about this stage is that it is active media consumption, i.e. highlighting and taking notes as you read.

Here are some tips for getting the most out of the first pass:

  • Err on the side of more highlighting rather than less. It’s good to have a lot of context in your highlights. You can always eliminate things that are redundant or unnecessary in the second pass.
  • Highlight chapter titles. Titles provide useful context for organizing your notes and understanding the broader picture for an excerpt.
  • Prioritize the new, useful, and the interesting. There is no need for anything useless and uninteresting to exist in your knowledge base. Also, avoid including things you already know well.
  • Look for scaffolding. Keep an eye out for material that provides a “platform” to tackle more advanced concepts. For example, I always highlight definitions of important terms I’m unfamiliar with. I also like information about people, places, or things, since I can use these as context to learn related topics faster.

Technology I use for the First Pass

The tools I use at this stage vary depending on the type of media I’m consuming. There are two main criteria: i) the tool must allow me to record highlights and notes digitally, and ii) the tool must allow me to export materials to my note taking system (Roam) with little effort.

  • Books: Usually I purchase books on Kindle, which has excellent highlighting. I then use Readwise to automatically sync highlights to Evernote, and can copy and paste from Evernote to Roam (apparently Readwise now has a direct export to Roam, so it looks like I can cut out the Evernote middleman). For physical books, I use the Readwise mobile application, which has an OCR feature that lets you take a picture of the page you are reading and highlight it.
  • Articles / Blogs: I try to read all articles and blog posts on Instapaper, which allows me to highlight and take notes on articles. Again, Readwise allows me to import these notes into Evernote, which I copy / paste into Roam.
  • PDFs: At the moment I don’t have a good solution for PDFs. Since I can’t highlight them easily, I’ll often just take notes directly to Roam as I read, with the PDF open in one window and Roam open in another.
  • Videos: Typically I’ll sit at my computer when watching videos. So, I keep Roam open while watching the video and take notes, with timestamps. Here’s an example of video notes I put together for a Jeff Bezos Lecture on innovation.
  • Podcasts: I do listen to some podcasts (especially Conversations with Tyler), although I haven’t found a great tool for taking podcast notes. I usually listen to podcasts when I’m on the go, so note taking in the browser is typically not possible. Usually, if there’s a podcast that I listen to that’s really good, I’ll just revisit it and take notes while I listen at my computer desk.

Second Pass: Consolidate and Summarize Notes

In this pass, edit the highlights and notes that you’ve exported to your note-taking system. Activities here can include:

  • Deleting irrelevant or redundant notes
  • Summarizing excerpts into key points, while keeping some direct quotes from the material that is notable or quote-worthy
  • Reformulating material into your own words
  • Bold-facing the most important points so your notes are “glanceable”
  • Creating commentary on the material, expanding on points that you liked, critiquing points you disagreed with, filling in missing arguments, creating connections to other material in your knowledge base, or creating a high-level “book-review” style summary
  • Marking particularly important material for long-term memory

A key advantage of the second pass is it produces a valuable digital asset that you can draw on the rest of your life: you now have a searchable, quick-to-read summary of what you have read. The knowledge is now part of your “second brain” for easy access and connection to your existing knowledge.

Yet another advantage of this second pass is that the act of editing helps you absorb the material. This is because it requires elaboration and recall, which are both well-known to foster learning and memory.

Technology I use for the Second Pass

My note taking tool of choice is Roam. It is a fantastic piece of software, although it’s difficult to explain its value in words (you really just have to try it). I recommend looking into it if you are not already heavily invested in an existing note taking app. I find it allows me to easily make connections between knowledge, and its incredible functionality has led me to ditch Evernote, Asana, and 90% of Google Drive.

Third Pass: Commit to Long-Term Memory Using Spaced Repetition

In this third and final pass, add material material flagged “long-term memory” in the second pass to a spaced repetition system.

Spaced repetition is reviewing material at increasing intervals of time, allowing you to remember material with minimal effort. I won’t go into more detail here, but I highly recommend this overview by Gwern Branwen. You can also subscribe to my Spaced Repetition Newsletter.

The advantage of this third pass is access to your knowledge in the moment, when it matters, without any external note taking tools. This is useful for the many situations where it’s not feasible to consult your notes. For example: job interviews, meetings, or creative work where speed of thought is important and you need lots of in-memory scaffolding to make progress.

One nice feature of this third pass is you’ve thoroughly vetted the material in the first two passes. This means you’ll be less likely to add things you don’t need to your spaced repetition system, and you’ll only add information you understand (see rule 1 of flashcard knowledge construction: do not learn what you do not understand).

Technology I use for the Third Pass

Personally, I use Anki, which is probably the most popular spaced repetition software tool today. Some examples of other options include Mnemosyne, SuperMemo or even paper flashcards if you’re a technophobe.

Examples of the Triple-Pass System in Action

To get a feel for what the end result of this system looks like, here are a couple of examples:

This Seems like a lot of Work…

It’s true that using this system will almost certainly mean you’ll take more time to consume media. Compared to just passively reading a book, the highlighting, summarizing, and spaced repetition all add time.

But before you dismiss it, ask yourself a couple questions.

First: are you a genius that effortlessly absorbs the materials you consume? I don’t mean this sarcastically. People like this exist, like Tyler Cowen. If yes, there’s no real benefit to you from this system. It would just slow you down.

Second: why are you consuming the media in the first place? Is it something you really want to remember? If the answer is yes, then I believe using a system like this is a no-brainer.

Yes, it takes some extra time. But if you are consuming high-quality material relevant to your life, the benefits of that extra 10-30% of effort is well worth it.

For access to my shared Anki deck and Roam Research notes knowledge base as well as regular updates on tips and ideas about spaced repetition and improving your learning productivity, join “Download Mark’s Brain”.

Tips From Anki Flashcard Refactoring: Add Enough Knowledge to your Deck and Review your Sources

My flashcard refactoring for today is a reminder of the classic knowledge construction advice: do not add what you do not understand. It is also a reminder of the importance of providing enough related cards in your deck for a piece of knowledge.

Here’s the card I came across that was giving me trouble, related to SQL programming (double-sided):

  • Side 1: Oracle SQL syntax for creating object table
  • Side 2: CREATE TABLE (table name) OF (object type)

When revisiting this card, I realized that I didn’t have a good concept of what “object tables” are, so this is definitely a case of not understanding the material before committing it to spaced repetition.

But the thing is, I wouldn’t have added it if I didn’t have a good understanding of object tables, at the time of adding knowledge to my spaced repetition system. The problem is I forgot the concept of “object tables”, and seeing the answer to this card was not enough to bring it back. I didn’t have any other cards in my deck about “object tables” and how they differ from other related concepts in Oracle SQL such as nested tables.

In a situation like this, it helps to go back to the source, clarify any misunderstanding, and add new cards that solidify your knowledge.

So, in this case, I looked up Oracle documentation and found a great article almost immediately that clarified the meaning. It also provided a bunch of useful nomenclature for closely related concepts, providing further scaffolding for the knowledge. This lead me to add a bunch of cards:

  • Card 1 (Cloze): Objects can be stored in two types of tables: [object tables] and [relational tables].
  • Card 2 (Basic 1-sided Q&A):
    • Q: What’s the difference between object tables and relational tables? (Oracle SQL)
    • A: Object tables store only objects Relational tables store objects with other table data
  • Card 3 (Basic 1-sided Q&A):
    • Q: What does each row represent in an object table? (Oracle SQL)
    • A: An Object

So to recap, here the main lessons from this refactoring:

  1. Don’t add stuff to spaced repetition that you don’t understand
  2. Make sure you add enough knowledge about the concept in your deck, so there is sufficient context for you to understand again when you forget
  3. When dealing with 1 or 2, the solution is to go back to the original source to understand the knowledge and add more relevant material.

For access to my shared Anki deck and Roam Research notes knowledge base as well as regular updates on tips and ideas about spaced repetition and improving your learning productivity, join “Download Mark’s Brain”.

Tips from Flashcard Refactoring

Include your Sources, Have a Single Answer, and Break-Down Your Cards

Here’s a flashcard related to Oracle SQL that was giving me trouble (lapsed 8 times and was automatically marked as a leech):

  • Side 1: Collection (Oracle SQL)
  • Side 2: Data types in Oracle SQL that lets you internalize parent-child relationships between tables in the parent table.

This was a double-sided card, so both Side 1 and 2 serve as the question. Let’s see if we can improve this one.

First things first: do I need this card at all? Yes: SQL is highly relevant to my career in Data Science, and the organization I work for relies heavily on Oracle database. It’s important knowledge for me that I didn’t want to remove.

Next, figure out the issue with the card. Looking at the card statistics, it turns out I was always getting Side 2 wrong. After some consideration, I realized that this is actually a poor definition of a “Collection”. In fact, it’s not really the “definition” of a Collection, but a characteristic of a Collection. In other words, the flashcard doesn’t have a unique answer: it’s true that a Collection internalizes parent-child relationships, but it does a lot of other things too.

I consulted the original source of the material and there isn’t a clear definition of a Collection there. I did some Googling for other sources and apparently there isn’t really a great definition of an Oracle Collection. It turns out that Collection refers to a generic programming idea not specific to Oracle.

So, rather than trying to define Collection, I’ve opted to break the existing card down into multiple cards, following Rule Number 4 of Knowledge Construction: stick to the minimum information principle, which means if you can break a card into multiple simpler, easier-to-answer cards, do it.

Card 1 (one-sided):

  • Side 1: What Oracle SQL data type lets you internalize parent-child relationships in the parent table?
  • Side 2: Collection

Card 2 (one-sided):

  • Side 1: What kind of relationship does an Oracle SQL Collection help you represent?
  • Side 2: Parent-child (aka “one to many”)

Card 3 (one-sided):

  • Side 1: Does the Oracle SQL Collection data type internalize parent-child relationships in the parent table or child table?
  • Side 2: Parent table

I also tracked down a good definition of the generic “Collection” concept in Computer Science, and added it:

Card 4-5 (double-sided):

  • Side 1: Collection (Computer Science)
  • Side 2: Object that groups multiple items together as a single unit (Computer Science)

I feel confident these cards will be easier to remember, cost less time and frustration, and help me remember the concept much better.

Lessons learned:

  • Flashcards should have a single answer. Multiple correct answers for a card is a recipe for confusion and frustration. Interestingly, this isn’t included in Poitr Wozniak’s Twenty Rules for Formulating Knowledge, although you could interpret this as a form of interference (Rule #11)
  • Keep track of your source material when making cards. It makes it easy to look up more details when needed. 
  • Browse related sources through Google search if you’re unsure about what to do to an item. This will give you more context around the card to see whether the knowledge is even required at all. You may also come across a clarification or better formulation. In the example above, I discovered the generic concept of “Collection” in programming and realized that it was futile to try to include a definition specific to Oracle SQL.
  • Break cards down into a larger number of simpler cards. This is classic knowledge construction advice that is often not heeded, because it feels like more cards means more work. Counterintuitively, it is really a free lunch: you remember the concept better, you spend less time reviewing than you would have with the single complicated card, and reviews become much more enjoyable. 

Combating Knowledge Interference (Flashcard Refactoring)

I came across this computer networking Anki flashcard I’ve forgotten over 7 times:

(NW for Sysadmins: Ethernet) Address Resolution Protocol (ARP) – Maps Ethernet addresses to IPv4 addresses and back.

The card uses cloze deletions [] like this:

  • (NW for Sysadmins: Ethernet) [Address Resolution Protocol (ARP)] – [Maps Ethernet addresses to IPv4 addresses and back].
  • (NW for Sysadmins: Ethernet) Address Resolution Protocol (ARP) – Maps [Ethernet addresses] to [IPv4 addresses] and back.

Interestingly, I haven’t forgotten a single review for the right-hand side clozes. Turns out this was the one causing me trouble:

(NW for Sysadmins: Ethernet) […] – Maps Ethernet addresses to IPv4 addresses and back.

Why? The issue seems to be another card in my deck that is very similar, and I’m confusing the two. The other card quizzes a networking concept called “Neighbour Discovery (ND)”, which performs a similar function to ARP except it maps IPv6 addresses to Ethernet and back rather than IPv4 addresses. This is a good example of interference, which refers to the fact that learning similar things can make you confuse them (see Rule 11 of Poitr Wozniak’s classic article on the 20 rules of formulating knowledge).

So the solution I’m opting for is pretty simple, just add a hint:

  • (NW for Sysadmins: Ethernet) ARP (hint: not ND) – Maps Ethernet addresses to IPv4 addresses and back.

One other small improvement is adding another card for the acronym alone:

  • Front: (NW for Sysadmins: Ethernet) ARP (Unpack Acronym)
  • Back: Address Resolution Protocol

These interference issues are tricky because you can’t really anticipate them in advance. You have to discover them as you review your cards.

Another annoyance is I’m not 100% sure that interference was actually the problem. Ideally, I would have discovered this troublesome card during review, so I could know for sure why I’m failing.

So here are some lessons learned from this little exercise:

  • Use hints as an effective tool for reducing interference.
  • Keep an eye out for interference during review of your knowledge. As soon as you encounter it, note it. In the case of Anki, there is a “mark card” feature. I also recommend actually writing text within the card to remind yourself exactly how you failed the card when you fix it later. It would be nice to be able to see basic card statistics, like number of lapses, during review without having to go into card statistics. I inquired on reddit whether there was an addon for this and while there are some good options for desktop, it doesn’t seem like there’s anything that quite meets this need for mobile (where I do all of my reviews).
  • As you get better at knowledge building, interference will become your most common problem. As Poitr Wozniak says, interference is “probably the single greatest cause of forgetting in collections of an experienced user” of spaced repetition systems since it is hard (impossible?) to avoid even if you are really good. You typically discover it during knowledge review time, not knowledge construction time.

Flashcard Refactoring

I’ve started a weekly habit of flashcard review. I want to share with you my thought process for modifying my cards, because I think this will be valuable to help you improve your own knowledge construction skills.

I also want my flashcard development out in the open so you can call me out when I make mistakes and provide suggestions for further improvements. Please do reach out! I am by no means the ultimate expert in knowledge construction.

So, I will be doing a regular series I call “Flashcard Refactoring” (Refactoring comes from the programming term which basically means revising and improving your code).

To sniff out poor flashcards, I ran prop:lapses>7 in the Anki browser to get all the cards I’ve forgotten over 7 times. Here’s one I came across about a command in the Linux command line to a suspend a job:

  • Side 1: ^Z (Linux Command Line)
  • Side 2: Suspend a job running in the foreground (Linux Command Line)

The card is reversible, so are two cards in total: one with Side 1 as the question, and another with Side 2 as the question.

At first glance, it doesn’t look too bad. It’s fairly concise. But one quick and easy change is reduce words in Side 2, in accordance with the 12th rule of Formulating Knowledge (“Optimize Wording”):

  • Side 2: Suspend foreground job (Linux Command Line)

This is a nice little improvement, but why am I really forgetting this card? I think it’s because ^Z doesn’t really have any meaning – it seems arbitrary and it has no clear connection to suspending foreground jobs.

So, I’ll create a fake connection, i.e. a mnemonic.

The mnemonic that immediately came to mind was the fact that the beginning of “Suspend” kind of sounds like a “Z”, e.g. “Zuspend”. I think this is all that’s required for this to stick in my memory (but only time will tell).

When you come up with a mnemonic, it’s a good idea to create a separate card for it, so I added the following to my deck: 

  • Q: Mnemonic for remembering ^Z suspends foreground job in Linux Command Line.
  • A: Zuspend

Anki / Spaced Repetition Tip: Review your Weak Flashcards

I’ve been a long-time user of spaced repetition tools. I’ll never forget first hearing about SuperMemo from a close friend as I started my undergraduate degree in 2005. I was immediately sold on the value of spaced repetition, and I particularly liked the idea of computers automatically taking care of review scheduling for you. I started using SuperMemo as a central tool for studying, and saw my academic performance skyrocket.

Over the years, I’ve slowly improved my skill in designing flashcards. It is by no means a trivial skill: it took me years to get pretty good at it, and to this day I still often make flashcards that are complete failures.

I believe there will eventually be an open collaborative platform for flashcard development and sharing, where experts can contribute and refine perfectly crafted cards. Users contribute their deck statistics, revealing poorly formed cards and contributing to our understanding of optimal flashcards.

But until that day, it pays to develop your flashcard creation skills.

Flashcard quality is top of mind for me since I’ve revisited the classic article by Peter Wozniak (of SuperMemo fame), “Effective Learning: Twenty Rules of Formulating Knowledge)”. It is a must-read for anyone that creates flashcards for learning (i.e. almost everyone at some point in their life). I’ve published my summary notes on this article (aside: my notetaking tool of choice is Roam – my notes are easy to copy-paste into your own Roam database if you happen to use it as well).

One great way to improve your flashcard development skills, while simultaneously improving the quality of your deck, is to review your old cards regularly. Review your top 10-20 most problematic cards weekly, and for each one you encounter, do one of the following things:

  • Revise: With the Twenty Rules of Formulating Knowledge by your side, refine your card or break it down into a larger number of small, easy to digest cards.
  • Suspend: If you don’t think you need to have a card in spaced repetition anymore, but don’t want to delete it entirely, suspending is a good option.
  • Delete: If you know the knowledge is completely useless to you, trash the card entirely.

But what cards should you review? If you’re like me, you have a pretty big collection, and it’s just not feasible to review all your cards every week to find the weak ones.

Anki makes it quite easy to find these problematic cards. Two main search commands in the Anki Browser are useful here:

  • tag:leech – this finds all of the “leeches” in your Anki deck, which are cards that you keep forgetting. By default, Anki tags your card as “leech” when you fail a card 8 times.
  • prop:lapses>n – this reveals all of the cards you have failed (“lapsed“) over n times. You can set n to whatever number you like. Start with high-n cards and work your way down.

In addition to using these search techniques, I try to make a habit of “marking” cards that are problematic or poorly formed in some way, during review. If it’s an easy correction (e.g. obvious suspension, or small text changes), I’ll make the change right away in the mobile app. Otherwise, I will simply mark the card and filter it out during weekly review to make improvements.

When you do revise your cards, I recommend “resetting” the card so it’s like a “do-over” – the card should be reviewed again as if you just created it. This serves two purposes: it ensures that the card will no longer show up in your “problem cards” lists when you do the above queries. It also provides you with more opportunities to review your new formulation of the knowledge.

Unfortunately, it seems the only way to do this in Anki is do create new card(s) with the information you want and delete the old one. There is an option for “rescheduling” the card, but this only restarts the review process and doesn’t delete your review history. As a result, the card will still appear as one of your problem cards if you do a query like prop:lapses>n. Luckily, it’s not much extra effort to do this.

I have to admit that I do not entirely practice what I preach here. Weekly review of my cards is something I haven’t fully incorporated yet, but I’m resolving to start doing it today. In the next weeks, I’m going to experiment with a Flashcard Refactoring series to illustrate the card refinement process. Stay tuned!

How to Deal with Lists in Anki

Photo by Glenn Carstens-Peters on Unsplash

When adding cards to Anki (or any other spaced repetition system), you’ll find you often need to deal with lists. In the spaced repetition community, these are sometimes called “Sets” (see Anki Essentials, Chapter 7, “The 20 Rules”).

For example, here’s a section from a book I’m working through called The Art of Doing Science and Engineering by Richard Hamming, where he summarizes the main advantages of computers over humans:

“People are sure the machine can never compete, ignoring all the advantages the machines have…These are: economics, speed, accuracy, reliability, rapidity of control, freedom from boredom, bandwidth in and out, ease of retraining, hostile environments, and personnel problems.”

This is a frustrating passage, because it’s packed with valuable information but tough to turn it into flashcards.

One easy approach is to cloze delete each individual option, and then cloze delete the entire list to make sure you can recall the whole thing. While this makes the cards easy to create, you’ll run into these issues when reviewing them:

  • It’s a lot to read, and you have to read every option to know what the missing one is.
  • It’s hard to remember. Even if you do individual cloze deletions for each point, you will still be very likely to fail on the question where you are asked to recall the entire list. Also, many lists (including the example above), are open ended and not necessarily exhaustive. So, when it comes time to review, you’re not quite sure what’s on the list, and you may list things that are “correct” but just not included on that particular list.
  • It violates a fundamental rule of flashcard creation. The “Minimum Information Principle” states that your questions and answers should be as simple as possible. A large number of simple cards is far more efficient to remember than a small number of big, demanding, awkward cards.
  • It’s hard to rate yourself. How do you rate your performance when you get 9 out of the 10 answers in the list correct? Do you fail yourself if you only got one wrong?

So, what are our options for dealing with these annoying lists?

Break it Down

First, ask yourself: do I really need to memorize the list? Do you really need to be able to spout off the list of advantages of computers over humans, and not miss any of points? Probably not.

Then ask yourself, what do I really want to get out of this list? In my example, I want advantages of computers over humans to pop up in my mind when the time is right. For example, when I’m working on a process at work that is repetitive, rote, and requires accuracy, I want to recognize computer scripts as a good solution.

So, I would not Ankify this specific list at all. Instead, I would look in the source to Ankify material focused on each individual point. For example, I could add specific questions on the relative costs of computing compared to manual effort, or details about the speed of computing compared to hand calculations. Doing this will give a deeper understanding of the advantages of computers over humans. It will probably also increase the likelihood of being able to recall the entire list at once (even though it’s unlikely I’ll need to do that).

You can also lists into sub-categories and then add those smaller lists to spaced repetition. Aim for each sub-category to have 1-2 items.

Take Advantage of Special List Structure

Sometimes you can take advantage of the inherent structure of a list to break the flashcard down. For example, your list can take a special form of a “1-n relationship”, as discussed in this thread .

A 1-n relationship is best understood by example. Suppose you want to memorize the Provinces in Canada. This list has the special property where each province in the list is associated with one and only one country (i.e. 1 country – n provinces). With lists like this, you can add questions like “what country does the province of Manitoba belong to?”, “What country does the province of Alberta belong to?”, etc. Of course, this fails if there is a province in another country with the same name, (that would mean it’s not a 1-n relationship), but you get the point.

Turn the List into an Enumeration

One of the problems with the example list of computer advantages is the order of the items has no meaning. There is no reason why “personnel problems” should be last rather than first. This makes it far more difficult to memorize because it’s just a bunch of seemingly random points.

Enumerations are ordered lists. Although they can still be difficult to memorize, they at least have some structure that your mind can latch on to. You should always try to convert your list into an enumeration if it’s not one already.

Ideally, your ordering will contain useful knowledge itself. For example, you could list the advantages of computers over humans, from most important to least important according to the author.

There are many other possibilities for turning lists into enumerations, depending on what you are trying to memorize, such as chronological order, area of the body, size, popularity, population, etc. If you can’t find a meaningful ordering, you can always list items alphabetically.

Enumerations are a free lunch, because they make list easier to memorize and provide bonus knowledge about the material. For example, wouldn’t it be much better to know the order of the planets in the solar system from closest to the sun to furthest, rather than just knowing the individual planets?

Use Cloze Overlapper

After you have turned something into an enumeration. How should you design your repetitions?

One option is to do one cloze deletion per item in the list. For example, take the list (a, b, c, d, e, f). Your cloze deletions are then ([…], b, c, d, e, f), (a, […], c, d, e, f), (a, b, […], d, e, f), etc.

I have done this many times in the past, and while I can usually answer the questions quickly and accurately, I find it hard to retain the information. When asked to reproduce the whole list, I fail. This is because the questions are too easy – removing only one item provides way too much context.

Until recently, I thought this was the best option. Then I came across the “overlapping cloze”: close deletions on individual items in the list, but with only a small amount of context given. For example, something like this: ([…], …, …, …, …), (a, […], …, …, …), (…, b, […], …, …), etc.

It’s actually quite painful to create overlapping clozes in vanilla Anki. Luckily, there is an add-on for this: Cloze Overlapper. This video by Glutanimate, the author of the add-on, provides an overview of issues with lists in and illustrates how how to use the add-on (aside: the author is the developer of another great add-on, image occlusion enhanced). I haven’t tried out Cloze Overlapper yet, so I can’t fully vouch for the technique or the add-on, but it seems to be highly regarded in the spaced repetition community.

Use Mnemonic Techniques

With a bit of practice, mnemonic techniques can help you achieve incredible feats of memory. Many are easy to learn. Even doing a little bit of studying on these techniques can produce big payoffs for your memory.

The technique I probably use most is the link method. For example, one flashcard I have on computer networking asks me to recall the main types of datalink layer errors. I added it as an unordered list: frame errors, drops, collisions, and overruns.

It’s actually a pretty poor card, but I haven’t had much trouble memorizing it because of the mnemonic I used:

  • I think of Link (from Zelda) with a shirt that says ERROR on it (dataLINK layer ERRORS).
  • Then I imagine him putting on some glasses (frame errors)
  • Link throws the glasses down to the ground (drops)
  • A giant truck hits Link (collisions)
  • Finally, the truck runs over the glasses that are on the ground (overruns)

Sounds silly, but it works. Our minds are extremely good at retaining images.

I am by no means an expert in these techniques, but I’ve found that learning a little bit helps a lot. If you go deeper, I’m sure you’ll see even bigger benefits. Other useful memory systems include peg and Method of Loci.

If you use a mnemonic technique for a list in your spaced repetition system, be sure to add flashcards to remember mnemonic itself.

Conclusion

Avoid lists if you can, but if you must, try to break them down into sub-categories, make sure they’re ordered (preferably with some relevant meaning to the ordering), use cloze overlapping to implement the reviews and limit context, and use mnemonic techniques. Together, this will give you a solid arsenal for committing lists to memory without much pain.

Resources