Spaced repetition in natural and artificial learning

Posted on by
Reply

In the last installment, I argued that sometimes we need to learn (gotta start easy). In a couple of Quora answers, I’ve been building up how we do that: an overview of effective learning techniques and a deep look at perceptual chunks and deliberate practice. What I’m slowly working toward is a philosophy and framework for designing learning tools. Today I want to talk more about a core tenant of this philosophy (hinted at in the last paragraph of my deliberate practice answer as well as a weekly review about Shuff’s philosophy): we are naturally good at learning within our local environment. However, we are increasingly living in a global, disconnected, artificial world that requires artificial environments to act and learn effectively.

Consider the example from Design vs. learning: my concern about my friend’s choices about bottled water was not about perceivable effects in the local environment, but rather the effects from its manufacturing halfway around the world. Rather than naturally learning within a responsive external representation, we require building up a mental model and evaluating decisions by running the effects of an action on that model. As my friend illustrates, this takes more work than most people will do, even with good intentions.

Let me revisit my favorite learning topic, spaced repetition, with this philosophy in mind. Scott H. Young and Khatzumoto are two of my favorite bloggers that write about learning. Several years ago Khatz introduced me to spaced repetition. To say it left an impact is an understatement. Since then I’ve made a series of projects that use spaced repetition in online learning systems. But I’ve always questioned when exactly spaced repetition is a good design choice for a learning tool.

Scott’s MIT Challenge is his attempt to study a four-year MIT curriculum in one year using online resources. He recently posted a great article questioning the value of spaced repetition for his own learning. He argues that aggressively pressing forward in learning new and more advanced material will naturally re-expose him to material from before, making a spaced repetition system unnecessary.

Such a result has been found for an elementary math curriculum. In Why Students Don’t Like School, Daniel Willingham (another of my favorite learning bloggers) summarizes the results of a longitudinal study: “A student who gets a C in his first algebra course but goes on to take several more math courses will remember his algebra, whereas a student who gets an A in his algebra course but doesn’t take more math will forget it. That’s because taking more math courses guarantees that you will continue to think about and practice basic algebra.”

Enter Khatzumoto. In his Unified Reading Process, he uses decks in a spaced repetition system as a collection system for everything interesting that he encounters. He lists thirteen examples of decks that he uses!

Some things that humans learn fold out naturally. Our brains co-evolved with the environment to let that happen. When we are born into the world, that world has, for hundreds of thousands of years, been one where we are expected to be in a society of people who talk to us as we naturally learn language and how to attract and care for other people and in an environment where learn how to navigate and hunt.

A good curriculum teaches you an artificial subject in a similar manner. It’s like a game that’s carefully set up to advance in difficulty as you use the skills and equipment you’ve gathered on the way. Subjects like math and science may be artificial, but curricula have undergone thousands of years of refinement to be somewhat learnable. This is what Scott Young is relying on when he presses forward with the MIT challenge.

But Khatzumoto’s method is setting up an artificial world where one doesn’t exist, where not even a good curriculum exists. He’s building his curriculum in place. Topics like foreign languages have some options for curricula, but why not make them contemporary and interesting by using real media? There’s no curriculum for being up to date with the latest trends in business or software engineering, but it’s important: you need to be able to converse with others in that vocabulary, and you may pick up some wisdom along the way.

Spaced repetition specifically replicates some of the advantages of a natural environment. Memory works like this: when we encounter important things a number of times in different contexts, we begin to learn them in the abstract. Otherwise, we’d be totally overwhelmed by the number of abstract concepts we could apply in any circumstance. By artificially spacing repetition, we allow the context to vary via the passing of time. Not only do our physical surroundings change, but the knowledge we have that can be related to what we learn changes.

So I don’t agree with the extent to which Scott thinks aggressive learning makes spaced repetition unnecessary. Moreover, he’s overlooking the “spaced” part of the concept: that not repeating, and allowing the context to vary, is an equally important part of the equation. But there is an art to keeping that environment spruce: Khatzumoto says, “I choose decks in order of fun/priority and I delete extensively. If I’m avoiding a deck, then I go on a deletion spree, and I keep deleting until the deck feels good again.” More on that to come!

Design vs. learning

Posted on by
2

The angriest I’ve gotten in recent memory is when arguing with a friend about her bottled water drinking habits. It wasn’t that she drank the water that made me angry but that she didn’t want to consider any information that might suggest why she should or shouldn’t drink (as much) bottled water (if you’re scratching your head about why this may even be an issue, see for example http://science.howstuffworks.com/environmental/green-science/bottled-water4.htm). Her argument was along the lines of “I know how behavior change works. Information won’t lead to change in my behavior.” This is fascinating because it’s a failure in rationality that results from a misunderstanding of theories about failures in rationality, which have come to the attention of someone like her only recently due to the proliferation of behavioral economics.

Her reasoning is rooted in empirical research. One example of many from Nudge is where Minnesota taxpayers were given different types of information about complying with tax law. Only one group had a significant change in behavior. It was the one given a not-so-informative social cue, simply that “90 percent of Minnesotans already compiled, in full, with their obligations under tax law.” The generalization abstracted from this and many similar studies is that various social and perceptual cues are far more effective than information in producing a desired change in outcome. As a generality, I think it makes sense.

Consider this analogical experiment, which sounds like a horrible word problem come to life. A group of fifth graders were taken to a store that was having a sale. Two identical shirts were for sale, but one was 60% off while the other was 30% off with an additional 40% off the sale price. One group of children was given information about how to multiply percents. Another group of children were not given the information but rather the sign for the cheaper shirt was shown in bright colors. The result (OK, not a real experiment, but a reasonable guess): many more children in the second group got the cheaper shirt! So information sucks, right??

There are two problems with this conclusion that are much easier to recognize than the tax compliance one. First, the information is not properly presented. Fifth graders are not able to understand and apply math with percents after the presumably brief intervention provided by the experiment. The second problem is that the second group of children happened to be nudged to the cheaper shirt[1]. Generalizing this idea of “let us be nudged” relies on some unnamed party to have the good intentions toward the nudgee, the right idea about how to nudge, and so on.

When we can understand and apply information, we become more powerful and free. If we know some math, we can calculate that the first shirt is 40% of the original cost, while the second is 42%, so the first is cheaper.

There is a limit to what we can know. We may be able to, by ourselves, calculate the better deal in a store. We cannot calculate the aerodynamics of the wings of an aircraft before hoping on board. We trust our lives to the engineers, the pilots, and the air traffic controllers when we fly on a plane. Even what we learn is trusted to the planning of curriculum designers, school board members, and teachers.

Back to the bottled water. My friend, who drinks the bottled water at work, was in some sense nudged into this habit by the free and readily available water. She is even nudged into some sense of environmental responsibility due to the recycling efforts at her workplace. But assuming that she has access to tap water and not an overly biased perception of the taste of tap water[2], there is little barrier to amending this habit.

Imagine my friend was a thoughtful and steadfast environmentalist and, somehow, wasn’t aware of any possible negative consequences of consuming bottled water. I imagine that she would have quickly devoured the information I presented and took action to change her behavior. But if she doesn’t have that disposition, she will not simply act on the information alone. Like the fifth graders at the store, she must learn. If she is to be convinced to change her behavior on this issue, she must learn facts about the effects of bottled water, and, more importantly, beliefs about the importance of the issue[3]. But learning is difficult and time-consuming, and most people devote little if any conscious time to learning.

At one point she pointed to the company. “If I shouldn’t be drinking bottled water, they should do something about it.” But who is “the company”? Practically speaking, it’s probably the office manager who’s in charge of what is made available in the office kitchen. But even the office manager may get passed down instructions on what to offer based on central planning for all office locations.

My friend, empowered with knowledge, could try moving up the chain, talking to the office manager and then to the manager’s manager. In a company like hers, it’s likely that the office manager has a bit of flexibility. In the most positive front, there seems to be a movement in design towards allowing decisions to be made locally by end-users[4]. But even if my friend is designing the office kitchen herself, before (and maybe even after) her argument with me, she would choose the bottled water. First she must learn.

EDIT 7/27 – changed the wording and added footnote about what she “must learn” since multiple people were confused.

[1] There is a middle ground between knowing math and nudging someone directly to the answer, which is better tools to do (“truthful”) math without a full understanding of the underlying concepts. I’ve discussed this in past weekly reviews in relation with Bret Victor’s Kill Math project. It didn’t fit into this post, but maybe I’ll bring it up again.

[2] “But a couple of very non-scientific, blind taste tests have found that most people — or most people in New York City, to be more accurate — can’t actually tell the difference between tap water and bottled water once they’re all placed in identical containers.” http://science.howstuffworks.com/environmental/green-science/bottled-water3.htm

[3] I think what was finally a bit convincing for her was the fact about how much water and fuel are used in manufacturing of the bottle, which is not offset by recycling it. I estimated that her consumption over a year could perhaps be enough to offset what someone needs to live, say in India.

[4] Co-design and metadesign are some terms for this. I plan to talk about this a lot more in the future.

Are mnemonics a waste of time for language learning?

Posted on by
Reply

Suppose you want to learn to write the (simplified) Chinese character 汉 (meaning: Chinese or Han). It’s made of two components, a 水 (meaning: water), represented as the three dots on the left, and a 又 (meaning: again), on the right. To remember this character I might remember a story using the two components and the meaning of the character: “Like a tide of water, dynasties like the Han have, again and again, risen and fell.”

Such mnemonic systems are popular. I have been following the “Heisig system” of the Remembering Simplified Hanzi books for a couple years. The question for this post is, is it worth it? I assume either way you are using a spaced repetition system in a standard way.

Scenario 1: You’re taking a class on Chinese, you might have tests where you are trying to remember a fixed set of characters, and you have a good amount of time to do so. You see “Han” on the page, think about it, and get a flicker of an image of a dynasty receding like a tide. “The tide–water… again and again… yes, that’s it!” The mnemonic certainly seems useful here, assuming you spent a shorter amount of time studying than without it (probably true).

Scenario 2: You’re learning Chinese over a span of several years. You want to be totally fluent in writing. Meaning that if you’re writing out an article, you don’t want to be conjuring up a story for every character and then translating that to components. You aren’t worried about knowing specific characters for quizzes in the intermediate stages. Which of the following is most accurate:

  1. A mnemonic is worth it in terms of how quickly you can remember a character. You can get fast enough with them that you don’t need to transition from “story translation” to automaticity.
  2. A mnemonic is worth it. As you become more and more fluent, you will transition to automaticity, and the mnemonic serves as a useful scaffold.
  3. A mnemonic is not worthwhile. Because you will eventually need to learn the character writing to automaticity, the mnemonic is simply an extraneous step.

I would doubt the first explanation because I no longer need to consciously recall the story for many characters. Although it could be that the story is being somehow used unconsciously. An even more extreme position would be that any memorization, whether you use a deliberate mnemonic or not, ties itself to stories and images in your unconscious!

The usefulness could be determined if you had many participants and several years for an experiment. The best I could find in a quick literature search was from Lawson & Hogben, 1997,

There is also support for the value of using deliberate
mnemonic strategies, particularly in the early stages of foreign
language learning (e.g., Carter, 1987; Carter & McCarthy, 1988;
Nation, 1990; Oxford, 1990).

“Early stages”, so already not what I’m looking for.

Finally, the concept of desirable difficulties (Bjork & Bjork, 2011) could lend support for the third theory, but I’m not clear on what conditions that is actually applicable.

Questions for DragonBox: Can algebra be taught with a game?

Posted on by
1
DragonBox is a mobile learning app that’s getting a lot of hype. Watch the video in that article or better yet try out the game for $3. I won’t talk about the obvious pro that DragonBox is fun and motivating. Instead, as a small exercise in learning science and experimental design, I’ll go through some questions that I would like answered to be convinced that DragonBox is actually succeeding at teaching people algebra:
  • Are most players able to advance through DragonBox?
  • Do the actions in DragonBox transfer to actual algebra problems?
  • Can students apply the constraints provided by DragonBox by themselves?
  • Is the procedural learning of DragonBox inferior to conceptual instruction?

Are most players able to advance through DragonBox?

This is the question that We Want to Know, the makers of DragonBox, should be able to answer easily by themselves by looking at the data of what people do in the game. It’s also perhaps the hardest question for me to judge because I’m entirely familiar with the underlying mechanism. It certainly feels like a typical puzzle game, where one can work up incrementally. Watching at five- and eight-year olds in that video, I’d would have to guess yes for most people.

Do the actions in DragonBox transfer to actual algebra problems?

Transfer is the thorn in education’s side. Ok, you’ve taught someone something, but can they actually use that in any context except the one they’ve learned? Often not. In a famous experiment by Gick & Holyoak, 1983, most subjects were not able to solve a analogous problem to one they were just taught. And in general, the evidence for effective educational games is incredibly sparse.

But here DragonBox does exactly the right thing–it gradually introduces the real symbols in place of the dragons and boxes. It seems to be a plus for DragonBox, but there’s one more issue:

Can students apply the constraints provided by DragonBox by themselves?

In DragonBox, when adding a monster to one side, you are not able to proceed until adding the same monster to other side (thus keeping the equation equal). I refer to this as an external constraint imposed by DragonBox. This constraint and several like it make it easier to stay on the right track when solving one of the puzzles. How much does this matter?

Prior experimental evidence says it may be a lot. Zhang & Norman, 1994 performed a number of experiments varying the type of constraints that were externally represented for the Tower of Hanoi puzzle. Some of these constraints made solving the puzzle drastically easier. Why? We have a limited capacity for thinking and the constraints make some of that processing automatic or nearly automatic.

But DragonBox may be different. The constraints here are rules applied in separate steps as opposed to constraints that affect the space of possibilities one has to consider. Although it may take a bit of work outside the game to really habitualize those steps, I don’t think that diminishes the value of the other parts that have been learned.

Is the procedural learning of DragonBox inferior to conceptual instruction?

I brought this issue up in my last post on Khan Academy, but it’s worth further discussion. DragonBox teaches procedural knowledge–it says nothing about the concept behind why a dragon on top of another of the same type becomes a 1. So even if learning from the game can transfer to real algebra problems, might it be better to use conceptual instruction from the start? Even though I don’t think the concepts really shine through, I do think the procedural knowledge gained is useful as an iterative part of learning.

Some representational changes that DragonBox may afford include:

  • Thinking of the equal sign as separating an equation into two balancing sides. (concept of mathematical equality)
  • Thinking of added quantities as loosely arranged terms, where the “loosely arranged” part may help understand them as commutative. (commutative property)
  • Thinking of multiplication and division as applying to each of those terms. (distributive property)
  • Thinking of negative terms as canceling. (additive inverses)
And there are also some ideas that ultimately fall more under the procedural umbrella, like the idea of canceling additive terms before multiplicative factors when isolating the variable.

Or… do an experiment

Confession: you don’t actually have to answer any of those questions to figure out whether algebra can be taught with DragonBox. You can just give people who don’t know algebra the game and see whether they learn it without any exposure to other instruction. You can give them a test before the game and a test after and see how much they learned.

In learning science, however, an experiment will typically use some type of control condition, a group that doesn’t use the game to learn. To see why, imagine that the group playing the game (called the “treatment group”) did improve. Maybe you accidentally gave an easier test at the end, or maybe they learned everything from the pre-test or watching cartoons or something. You don’t know for sure. So you would want another group (control group) that takes the same pre-test and post-test but doesn’t play the game in between. If the DragonBox group does better, you know you’ve (most likely) got something good!

Such a design would also let you do other types of comparisons, such as comparing the game to some other form of instruction, or using the game in combination with conceptual instruction and comparing that to either alone.

In conclusion, I’m cautiously optimistic about the ability for DragonBox to improve learning, especially if it is augmented with conceptual instruction. I’m also curious what other topics would lend themselves to a game like this (check out some discussion by Terence Tao), or alternatively what kind of crazy math I can now do after playing thousands of levels of Unblock Me.

Slow web yourself: how to send daily email from Google spreadsheets

Posted on by
4

Lately I’ve been thinking about the slow web. More on that later. I also started reading Tao Te Ching and, rather than speeding through the whole thing without absorbing much, I wanted to slow down my reading to one section per day, emailed to me each morning. I figured out I could do this, like most things, with Google spreadsheets. Here’s how:

  1. Create a new spreadsheet where each row has your email address in the first column and the text of the email in the second column.
  2. Open the script editor from “Tools > Script editor…”
  3. Add the following code: 
    function sendEmails() {
  var sheet = SpreadsheetApp.getActiveSheet();
  var startRow = 1;
  var numRows = sheet.getLastRow();
  var dataRange = sheet.getRange(startRow, 1, numRows, 2)
  var data = dataRange.getValues();

  var firstDay = new Date(2012, 6, 7);
  var today = new Date();
  var daysElapsed = Math.floor((today - firstDay)/(1000*60*60*24));
  var whichRow = daysElapsed % numRows;

  var row = data[whichRow];

  var emailAddress = row[0];
  var message = row[1];
  var subject = "Daily Tao";

  MailApp.sendEmail(emailAddress, subject, message);
}
  4. This script will march through each row in the spreadsheet day by day. Customize the starting date by changing the “firstDay” variable. The first value is the year, then it is the month minus one (July is 7, 7-1=6), then it is the day (so July 7, 2012 here). The “% numRows” makes it loop after getting through all the rows. You can remove that if you just want it to stop.
  5. Customize the message subject (currently “Daily Tao”).
  6. Now you need a trigger to run the script every day. In the script editor, go to “Resources > Current script’s triggers…”
  7. “Add a new trigger”. Change “From spreadsheet” to “Time driven”, then select “Day timer” and then choose the approximate time of day to receive the email. Press “Save”.

That’s it! You should be getting an email each day now. Another option is to send yourself a random message instead of going in order. Just change these four lines,

  var firstDay = new Date(2012, 6, 7);
  var today = new Date();
  var daysElapsed = Math.floor((today - firstDay)/(1000*60*60*24));
  var whichRow = daysElapsed % numRows;

to

  var whichRow = Math.floor(Math.random()*numRows);

Dear teachers: Khan Academy is not for you

Posted on by
2

Dear teachers: Thanks for what you do. But I have a message: Khan Academy videos are not for you. The videos are for students, and students are using them. So I think MTT2k is misguided. We should be sitting the students in front of the videos and trying to figure out what goes on in their head, rather than sitting the teachers in front of them.

Here’s why teachers won’t get it right: Expert blind spot refers to the idea that “content knowledge eclipses pedagogical content knowledge” (Nathan, Koedinger, & Alibali, 2001). EBS does not mean that teachers don’t have enough pedagogical content knowledge. It doesn’t mean that teachers (or researchers!) who know about EBS are suddenly able to think like a student. It means that when people think, they necessarily think using their content knowledge. Teachers cannot think like a student who does not have that content knowledge. Imagine a champion weightlifter just trying to imagine–with some degree of accuracy–how his barbell feels to a puny first-timer.

Your students are stacked on a motorcycle in the right lane.

The problem with MTT2k is that the teachers are trying anyway to imagine what a student is thinking when they watch one of Khan’s videos. Because teachers aren’t busy learning the material, they have all kinds of attention to direct at any detail that pops out without a complete, polished explanation. In the real world, we never have that luxury; we have to assemble our knowledge from incomplete, messy fragments.

The original–and still the best–Khan critique is Derek Muller’s commentary that Khan Academy does not address misconceptions in its videos. He compared a straightforward video introducing physics concepts to one that first introduced common misconceptions and then cleared them up by presenting the correct concepts. Although students found the first video clear and concise, they didn’t actually pay and attention and learn from it. (Wait, is that the same “clear and concise” that MTT2k producers are asking for?)

My point is not that teachers are wrong and Khan Academy is wrong and Derek Muller is right (just because we share a surname). My point is that we have to look at empirical data to determine what instructional styles actually work or do not work. So let’s answer some questions, shall we?

Can students learn from videos or even lectures in general? YES, with two caveats. The debate over direct instruction and discovery learning is long and brutal, but there are clear data that direct instruction can be effective[1], and Derek’s technique is one example of improving video instruction to overcome one thing that direct instruction opponents believe can’t be done with video.

Now, the caveats: one is that the student needs to be active and constructive when they are watching the video. The fact that they freely pull up a Khan Academy video is a good start. There’s no way to make sure this is happening 100% of the time, just like it won’t happen 100% of the time in the classroom.

The other is that students may be overestimating their confidence with the material[2]. In fact, I believe this is one of the major problems with Khan Academy’s videos. Khan, diligently working through every term expansion and long division, is just so good at making us watchers feel like we’re the ones doing the practice.

Should students start with concepts or procedures? Many students are educated without developing the kind of mathematical thinking that we mathematical thinkers would like them to have. This problem has often been attributed to a overemphasis on procedural learning in the classroom. But is the idea of starting with the concepts a form of expert blind spot? It’s a complex issue, and seems most likely that we need both to learn, depending on the exact topic[3]. Sal Khan is clearly interested in expanding Khan Academy’s conceptual video repertoire.

Should videos address student’s misconceptions? Sometimes. Derek Muller provides several compelling experiments where addressing misconceptions clearly improved performance over a straightforward. But all of Derek’s examples are areas where students typically have strong misconceptions that override their learning (kind of a fake expert blind spot). But sometimes students are really just learning something new and there are no real misconceptions to address. Sometimes they have even deeper problems that require a different approach[4].

My suggestion to begin approaching some of the problems raised above is to forget the flipped classroom, let’s flip Khan Academy. Let the practice be the guide. Students start with the practice and use it to figure out their weakness. Often, all a student needs is a flag on their error to be able to figure out the problem[5]. But not always. And then you can bring in the videos–in particular, the video that addresses exactly the incorrect or missing knowledge of the student. It is difficult but not impossible to assess the deep conceptual knowledge that we’d ultimately like to provide students[6]. And then Khan Academy can use real student data–not teacher’s rear view mirrors–to figure out which videos are not getting the point across.

Footnotes

[1]  The same Derek Muller has an excellent interview with direct instruction champion John Sweller. Klahr & Nigam, 2004 is one experiment where the direct instruction conditions outperforms a discovery learning. A stronger statement is provided by Mayer, 2004.

[2] Students, particularly low-ability students, are poor estimators of their ability level (Mitrovic, 2001). Re-reading a passage (and presumably re-watching a video) is a comparatively poor study strategy, but students tend to be believe it’s better than testing themselves (one of the best studying strategies) (Kornell & Son, 2009).

[3] Bethany Rittle-Johnson and colleagues have done a large body of work comparing conceptual and procedural learning (e.g. Rittle-Johnson & Alibali, 1999; Matthews & Rittle-Johnson, 2009). For learning decimal expansion, students used procedural and conceptual instruction in iteration to gradually build a better mental representation of decimal numbers (rittle2001developing).

[4] See Chi, 2008 for a few ways of classifying conceptual learning.

[5] See VanLehn et al., 2003 for a discussion.

[6] The Force Concept Inventory is a famous example of a conceptual assessment. It helped reveal that students who were scoring high on exams in a physics class weren’t actually learning the concepts from the lectures.

Weekly review through June 11

Posted on by
Reply

Pots and marshmallows

Bill Buxton has an oft-referenced story about a pottery class where one group was graded for their final pot and the other group was graded for the number of pots they produced (regardless of their quality). The second group made better pots — they were iterating and improving while the first was preoccupied with making the one perfect pot. Similarly, Tom Wujec talks about groups trying to build towers out of dry spaghetti and tape with a marshmallow on top. The most successful groups start with the marshmallow on top and iterate to increase the height of the tower. Other groups fail because they try unsuccessfully to add the marshmallow at the end.

The lesson I want to draw here is the importance of starting with the constraint and keeping it always in the picture. I believe that one way to drive a successful PhD career is to have a single good research problem that understands an important constraint in the world. That’s the constraint that guides all of your iterations. I’ve been thinking about this since coming across the biography Andrew Ko’s website, a very successful former HCII student. I noticed that his description of his voluminous work at HCII was a single problem: “trying to find out what made debugging so difficult, and inventing technologies to make it easier”.

My marshmallow

After thinking about it for a few weeks, I finally have a one-liner that I like: “How can we make the learner come back tomorrow?” Seems simplistic, but there are a few reasons I like it:

  1. There’s a user-centered value: if the user is coming back tomorrow, they probably like what’s going on.
  2. There’s a learning theory value: more than anything, learning takes persistance and consistency. We are naturally wired to learn, but there is a lot of information to absorb and skills to master.
  3. There’s even a business value: if the user is coming back every day, they’re probably going to be willing to pay or check out a sponsor or are at least looking at a lot of ads.
  4. A solution can be evaluated without requiring inter-rater reliability on some abstract construct from the 80s (but any papers submitted to CHI 2013 doing this should be accepted without question!) — literally just: did they try to learn more the next day?
  5. It doesn’t assume a technological solution.
  6. What may be part of a good answer is my favorite technological intervention, spaced repetition!
  7. As simple as this is, I think a lot of people working in intelligent tutoring systems aren’t asking this question. They may just assume the technology is going to be integrated into the classroom. Most experiments are looking at what students do in a single class period. The closest close-to-home example is Nudge, but that’s still relying on the motivational structure of a traditional course.
  8. It motivates some cross-domain work. Why does a gamer play every day? Are more substantial reasons than the addictive features of the game? Can we learn anything from why gamers quit (see also Koster, 2005)? Replace gamer with anyone serious about their craft. They’re all learning something.

A metaphor

A naive observer might conclude that the definition of a reader is someone who slowly, day-by-day, moves a bookmark from one side of a book to the other. They might not be too far off. Assuming someone knows how to read, when a book is well-written, the reader can just look at the words on the page and the brain does the rest fairly automatically. Millions of kids didn’t need comprehension monitoring strategies, concept maps, or highlighting to understand transfiguration spells in the made-up universe of the Harry Potter books.

Now imagine a book where the content is automatically generated with exactly what you need to learn. Reminders appear in the following pages just as you are about to forget what you’ve encountered, sections carefully repeat and rephrase what is difficult for you, assignments at the end of chapter suggest exactly the skills you need to work on. And a bookmark traces your progress as you come back tomorrow to move it a little further down. That’s what we can do with educational technology. That’s all we need to do.

Weekly review through June 3

Posted on by
Reply

Goals

Starting with reading The Power of Habit (Duhigg, 2012) several weeks ago, I’ve been trying to work on the habits in my life and the goals that guide them. I revisited Power of Full Engagement (Loehr & Schwartz, 2006) and completed the exercises to help figure out how to frame what is important to me. I also tried a more bottom-up approach of listing all of the goals, habits, and productivity tactics I might want to set or apply. This helped me draw out some things that were missed with the Power of Full Engagement process. I still need to work on consolidating them.

In addition to my personal goals, I’ve been thinking about how to design software that helps people set goals. I checked out Fitocracy, which does gamification and social features for fitness related goals. You can record your workouts and your friends gives props for your accomplishments. Another site that looks promising and more general is Goal Buddy. One of the insights I had was that goals are often set implicitly. For instance, in Stepmania, your scores are previous songs are recorded. The display of these scores implicitly sets goals for you to improve the scores. I wrote more about this at interface design for goals.

Weekly wiki

I added a visualization to my wiki’s front page that displays the amount of changes I’ve done with different pages over the past week. It turns out to have a number of nice affordances.

  1. I can see whether I’ve been doing as much work as usual.
  2. I can see which pages I’ve recently worked on — those are what I’m more likely to be working on at the moment.
  3. I can see which pages will soon fall off the first page in case there’s anything that I forgot I was in the middle of.
  4. I can do my weekly review very easily by scanning through the pages I accessed over the last week.

Someone suggested looking at it with longer term data as well, which may be very interesting.

Productive failure and a general theory of learning

I was intrigued reading about the productive failure effect a couple months ago in Kapur & Bielaczyc, 2012. Productive failure as defined in the paper is having students work on ill-defined problems that they can’t necessarily solve prior to regular instruction. What’s nice in the paper it is that they use direct instruction with the productive failure condition, so it’s a more direct test of this process rather than some broad instructional strategy. It’s surprising (to a direct instruction-leaning person) because the worked example effect seems to indicate the opposite: starting with fully worked examples is better. However, this prior failure theme has been demonstrated in papers like Schwartz & Martin, 2004, and Kurt Van Lehn has written about impasse learning, claiming that impasses are a necessary condition for learning (VanLehn et al., 2003).

Thinking about this led me to the following description of learning. Learning occurs when

  1. We experience an indication of failure.
  2. We have a particular reaction (brain response) to the failure.
  3. We store new information about how to correct the failure if this information is available.

#2 comes from the goal orientation work as described (with neuroscience!) in http://www.wired.com/wiredscience/2011/10/why-do-some-people-learn-faster-2/

I’ve found this little checklist useful for thinking through questions about what makes one learning intervention better or worse. For instance, what about direct instruction? Is it a problem because failure indications aren’t happening when someone is just talking to you? They can. I think Derek Muller’s video about Khan Academy (reflecting his thesis work) is a good demonstration of how to evoke this.

I’m not going to claim (yet) that these are the only conditions where learning occurs, and I think there are some points that need refinement, particularly what do the conditions of failure and of the corrective information need to be? Obviously tripping on a banana peel is not going to help me learn calculus.

Weekly review through May 27

Posted on by
Reply

Self-regulated learning framework

This week I spent a lot of time trying to find a framework for self-regulated learning that would work well for coding my data from interviewing self-studiers. I’ve compiled some stuff on a page called SRL model. I’m learning there are many ways to do a model or theory or framework or measure, and they have varying degrees of usefulness for my purposes. The ideal, as suggested by my advisor, is to have a model that basically shows the different states that a person can be in and then one can identify where there are breakdowns in the process. That’s the approach taken for the help-seeking tutor (Aleven et al., 2006, a diagram of the model), which led to improvements in an intelligent tutoring system in terms of getting people to use the help system in a way that was most beneficial to learning. The closest I could find from self-regulated learning is the model in Winne & Hadwin, 1998. Ideally, this model is also validated and empirically grounded in some way, i.e., the existence and distinctiveness of each stage can be demonstrated.

Other stuff

  • I read Confessions of a Public Speaker and collected some other advice on public speaking. I think the message is: practice your talk. I may even try recording my practices to get some authentic feedback. Which will make me vomit, but I think it’s worth it.
  • Read Clark & Mayer, 2011, which is an excellent overview of research that is important to designing computer-based learning interfaces. It’s also nice for free research questions when they go into questions that still haven’t been answered.
  • Curious what kinds of web-based learning interfaces are out there? I knew you were. Check them out: learning_tools:start.

Weekly review through May 20

Posted on by
Reply

Shuffour

I stopped using Shuff. My experience with Shuff has given me the following takeaways:

  1. I need to differentiate between standard to-do items (one-time tasks) and recurring tasks.
  2. The Do 5 system is nice for getting a chunk of one-time tasks done every day.
  3. Timeboxing is very often worthwhile. Especially combined with some data (even just casually observed) about how much can be accomplished in certain amounts of time.
  4. I want to have data that keeps me informed about my progress with things, but I’m not exactly sure what that is.
  5. URL lists are very handy for many of my activities.

For #1, I really like Habit List, which I just started today. You can specify all kinds of different frequencies (e.g. every three days, twice a week, etc.) for a task and then all of your tasks for the day will appear on a list. Each task has a calendar that shows when you’ve completed it, your streak, and so on, so that handles #4 in a new way that I like so far.

For #2, DailyDo.it is a nice clean interface, but it seems unnecessary to keep on top of Habit List. Maybe I’ll just use sticky notes with “Do 5″ as a daily task on Habit List.

For #3, I just open a tab with http://e.ggtimer.com/20%20minutes or whatever amount of time I want.

For #5, I put my “articles to read” and “general reading” lists directly on the front page of my wiki, which I’m opening a lot more now.

To combine all of them is the job for a hypothetical Shuff 4, which is essentially a mashup of Habit List and Google Calendar. The main interface is structured around a single day. Like Habit List, recurring tasks appear in a task list in one column. You can also add one-time tasks here. In an adjacent column is a calendar for the day (ideally synced with GCal). You can drag the recurring tasks onto the calendar and adjust their length of time. Finally, at the top of the page is the standard Shuff timer task name and timer countdown, but now it’s running automatically — going through your tasks in real time. It will also continue with the value recording feature from Shuff 3, which allows the data collection mentioned in #3.

After the amount of time I put into Shuff 3 only to quickly abandon it, I’m going to do a lot of thinking before moving forward with it. I encourage my blog readers to be extremely critical and let me know any foreseen problems with this.

Today I also ran into a really awesome app that basically functions as a Shuff-task-but-better for one of the most fearsome recurring tasks of all: checking email. The app is Email Game, and it turns your Gmail inbox into a fast-paced game. Maybe it’s a novelty effect, but I can’t describe how powerful it feels. I think the key may be that it bypasses the inbox list and having to focus on exactly one small thing. That’s kinda what Shuff was supposed to be, but there is a line between processing email and scheduling a whole day.

Individual differences in learning systems

Lately I’ve been referring to my research interests as DIY learning, after inspiration from Stuart Card’s talk at CHI mentioning DIY communities like Make and Quantified Self. My main interest in web tools is not just in how they can replace or supplement classrooms but how they can be employed for individual learning needs. I think that will involve things like self-tracked_learning, goal-setting, and a way of picking and choosing among tons of content.

I’m working towards some broad questions to get started: 1) to describe the space of possible interfaces, 2) to understand the individual differences that might influence how to build these. For the latter I was inspired by a blog post by Mark Guzdial about Turadg’s defense. I think it helps to divide up the individual differences into categories. Where the most controversy exists is probably that of “learning strategies”. In short, my claim is that it’s better to be system-driven than user-driven here. I’ve written some more on my wiki.