Weekly review through June 11

Pots and marshmallows

Bill Buxton has an oft-referenced story about a pottery class where one group was graded for their final pot and the other group was graded for the number of pots they produced (regardless of their quality). The second group made better pots — they were iterating and improving while the first was preoccupied with making the one perfect pot. Similarly, Tom Wujec talks about groups trying to build towers out of dry spaghetti and tape with a marshmallow on top. The most successful groups start with the marshmallow on top and iterate to increase the height of the tower. Other groups fail because they try unsuccessfully to add the marshmallow at the end.

The lesson I want to draw here is the importance of starting with the constraint and keeping it always in the picture. I believe that one way to drive a successful PhD career is to have a single good research problem that understands an important constraint in the world. That’s the constraint that guides all of your iterations. I’ve been thinking about this since coming across the biography Andrew Ko’s website, a very successful former HCII student. I noticed that his description of his voluminous work at HCII was a single problem: “trying to find out what made debugging so difficult, and inventing technologies to make it easier”.

My marshmallow

After thinking about it for a few weeks, I finally have a one-liner that I like: “How can we make the learner come back tomorrow?” Seems simplistic, but there are a few reasons I like it:

  1. There’s a user-centered value: if the user is coming back tomorrow, they probably like what’s going on.
  2. There’s a learning theory value: more than anything, learning takes persistance and consistency. We are naturally wired to learn, but there is a lot of information to absorb and skills to master.
  3. There’s even a business value: if the user is coming back every day, they’re probably going to be willing to pay or check out a sponsor or are at least looking at a lot of ads.
  4. A solution can be evaluated without requiring inter-rater reliability on some abstract construct from the 80s (but any papers submitted to CHI 2013 doing this should be accepted without question!) — literally just: did they try to learn more the next day?
  5. It doesn’t assume a technological solution.
  6. What may be part of a good answer is my favorite technological intervention, spaced repetition!
  7. As simple as this is, I think a lot of people working in intelligent tutoring systems aren’t asking this question. They may just assume the technology is going to be integrated into the classroom. Most experiments are looking at what students do in a single class period. The closest close-to-home example is Nudge, but that’s still relying on the motivational structure of a traditional course.
  8. It motivates some cross-domain work. Why does a gamer play every day? Are more substantial reasons than the addictive features of the game? Can we learn anything from why gamers quit (see also Koster, 2005)? Replace gamer with anyone serious about their craft. They’re all learning something.

A metaphor

A naive observer might conclude that the definition of a reader is someone who slowly, day-by-day, moves a bookmark from one side of a book to the other. They might not be too far off. Assuming someone knows how to read, when a book is well-written, the reader can just look at the words on the page and the brain does the rest fairly automatically. Millions of kids didn’t need comprehension monitoring strategies, concept maps, or highlighting to understand transfiguration spells in the made-up universe of the Harry Potter books.

Now imagine a book where the content is automatically generated with exactly what you need to learn. Reminders appear in the following pages just as you are about to forget what you’ve encountered, sections carefully repeat and rephrase what is difficult for you, assignments at the end of chapter suggest exactly the skills you need to work on. And a bookmark traces your progress as you come back tomorrow to move it a little further down. That’s what we can do with educational technology. That’s all we need to do.

Weekly review through June 3


Starting with reading The Power of Habit (Duhigg, 2012) several weeks ago, I’ve been trying to work on the habits in my life and the goals that guide them. I revisited Power of Full Engagement (Loehr & Schwartz, 2006) and completed the exercises to help figure out how to frame what is important to me. I also tried a more bottom-up approach of listing all of the goals, habits, and productivity tactics I might want to set or apply. This helped me draw out some things that were missed with the Power of Full Engagement process. I still need to work on consolidating them.

In addition to my personal goals, I’ve been thinking about how to design software that helps people set goals. I checked out Fitocracy, which does gamification and social features for fitness related goals. You can record your workouts and your friends gives props for your accomplishments. Another site that looks promising and more general is Goal Buddy. One of the insights I had was that goals are often set implicitly. For instance, in Stepmania, your scores are previous songs are recorded. The display of these scores implicitly sets goals for you to improve the scores. I wrote more about this at interface design for goals.

Weekly wiki

I added a visualization to my wiki’s front page that displays the amount of changes I’ve done with different pages over the past week. It turns out to have a number of nice affordances.

  1. I can see whether I’ve been doing as much work as usual.
  2. I can see which pages I’ve recently worked on — those are what I’m more likely to be working on at the moment.
  3. I can see which pages will soon fall off the first page in case there’s anything that I forgot I was in the middle of.
  4. I can do my weekly review very easily by scanning through the pages I accessed over the last week.

Someone suggested looking at it with longer term data as well, which may be very interesting.

Productive failure and a general theory of learning

I was intrigued reading about the productive failure effect a couple months ago in Kapur & Bielaczyc, 2012. Productive failure as defined in the paper is having students work on ill-defined problems that they can’t necessarily solve prior to regular instruction. What’s nice in the paper it is that they use direct instruction with the productive failure condition, so it’s a more direct test of this process rather than some broad instructional strategy. It’s surprising (to a direct instruction-leaning person) because the worked example effect seems to indicate the opposite: starting with fully worked examples is better. However, this prior failure theme has been demonstrated in papers like Schwartz & Martin, 2004, and Kurt Van Lehn has written about impasse learning, claiming that impasses are a necessary condition for learning (VanLehn et al., 2003).

Thinking about this led me to the following description of learning. Learning occurs when

  1. We experience an indication of failure.
  2. We have a particular reaction (brain response) to the failure.
  3. We store new information about how to correct the failure if this information is available.

#2 comes from the goal orientation work as described (with neuroscience!) in http://www.wired.com/wiredscience/2011/10/why-do-some-people-learn-faster-2/

I’ve found this little checklist useful for thinking through questions about what makes one learning intervention better or worse. For instance, what about direct instruction? Is it a problem because failure indications aren’t happening when someone is just talking to you? They can. I think Derek Muller’s video about Khan Academy (reflecting his thesis work) is a good demonstration of how to evoke this.

I’m not going to claim (yet) that these are the only conditions where learning occurs, and I think there are some points that need refinement, particularly what do the conditions of failure and of the corrective information need to be? Obviously tripping on a banana peel is not going to help me learn calculus.