Weekly review through March 25

Reviewing the daily review

I’ve been doing the daily review for over month now. So I want to check how things are going, and evaluate how it is working as a supplement to Shuff. Going through each item I’ve been tracking:

  • Learning Chinese with Skritter. It’s useful and reassuring to see my number grow. After tracking for a while, I made a projection to how many new characters I needed to add each day to get to 1500 without a reasonable amount of time. After that, I’ve been pretty consistent about studying each day.
  • Stepmania. I started tracking this about two weeks ago. I try to play the same song every day and track the change of “Flawless”-rated arrows. In contrast to Chinese, where my numbers increase just by showing up, I’ve had no noticeable progress. I’m not sure if that’s a property of myself or of the assessment — I’ll keep showing up for now and see how it goes.
  • Eating healthy. After reading Perfect Health Diet, I stopped trying to eat as much protein and realized a need to increase other macronutrients and think about micronutrients as well. I gave calorie tracking a shot, but I don’t think it’s worthwhile — the unknown about food from restaurants is too high, so I’ll just try to minimize those using the “happy meal” tracking. With the 80/20 principle, I should be getting about 16 good meals per week. Surprisingly, I have achieved that a couple of the weeks, but still need to plan my meals better.
  • Getting crap done. As suggested, I made a push to get my to-do list from a peak of 10 down to a low of 2, but now it’s gotten up to around 5 and stayed there. On the other hand, I think the inbox tracking has helped me keep that mostly low. One thing this suggests is to make the to-do list items easier to do, which should’ve been true from the start. For example, instead of “cancel bank account” (which I can’t bring myself to do thanks to the horrors of customer support), I could change it to something as simple as “write down number for Chase bank.”
  • Cleaning tasks. I previously said I was taking the “smaller step” of “recording whether I’ve done at least one “clean up” Shuff task that day.” In fact, think it’s a perfectly fine step. My house stays in pretty good shape doing just one or two during the week plus a bigger cleaning on Sunday.
  • Being a good researcher. I changed this from three items down to two: accomplishment and something random. Even two can be hard to come up with in a day, but it’s nice to have an accomplishment listed for every day (even though many of them are not research-related).
  • Going offline. Like this one. I realized how poorly I was doing one week and worked to fix it.
  • I also added location to keep track of how often I’m at school and the coffeeshop or when I’ve gone out to special events (rarely…), and a task to keep track of whether I did stretches.

Back to Shuff

A few of these are already tracked (whether I’ve cleaned) or trackable (whether I’ve cooked) by Shuff. Even then, they get lost in the noise. Some I really want to have the output number, such as number of Chinese characters learned, so that would have to be added to Shuff if I really want to takes things back there. It’d be fun to make a complete visual dashboard for all this kind of stuff together, but the more specialized I make Shuff the less likely anyone else will want to use it. Maybe.

Another thing I realized was that perhaps the Shuff philosophy of randomly interleaving small tasks is flawed. I noticed it’s really nice to have the dishes done and the inbox empty before I get down to work. Not always possible though.

Twenty questions with Evernote

I aligned Evernote to fit the idea of the research/personal questions. In the first week of Cognitive Perspectives to HCI, we talked about whether filesystems should be organized more organically — i.e. more associative. I think organization can also serve to help form a better mental structure. While I wouldn’t advocate eliminating the many natural associations we create, it’s probably also good to reinforce a structure that can be browsed or communicated coherently.

So basically, I have stacks which represent the different research questions. Within them I have notebooks that represent clusters of evidence to may provide a partial answer. For example, I have the question “How do we motivate people to pursue mastery-based learning?” with an evidence cluster “Use reflective assessment” that collects a few papers on the subject.

As I mentioned last time, it’d be good to incorporate with the rest of my tracking. Tracking the amount of stuff in each category might not be hard and would be cool to see. I think refining and revisiting are equally important.

Categorizing contributions

Daniel Willingham gave a great, entertaining talk on Friday. One point that he made are that education researchers have three ways to contribute to education: what must be done, how it could be done, and rigorous program-A vs. program-B results. The musts seem to generally come from basic psychology research. His example here is the need for factual knowledge to improve reading ability — it’s just how the brain works. How it could be done is more open but can be more applicable by teachers. Within learning technology, it’s not clear to me whether that should be providing example applications and evaluating them or producing design patterns or what.Anyway, it’s another thing to think about for categorizing my notes within the Evernote system.

Weekly review through March 18

Making learning interfaces is hard

Making software interfaces is hard — people like Alan Cooper have been writing (Cooper, 2004) and consulting about how bad things are for years and the status quo is still quite bad.

Meanwhile, there is endless controversy over how we learn. And completely modeling how we learn through artificial intelligence is what I’ve coined a HLAI-complete problem, that is human-level artificial intelligence-complete, meaning that if we were to solve it, we’d probably be solving a whole class of problems around human intelligence — and ringing in the singularity.

But that’s not to say we can’t do really cool things as an approximation. Google won’t understand your search query as well a human, but it can still often find you some useful websites based on the words you give it.

Neal and I have been working on a redesign for the real analysis material called Learnstream Rudinium. I hope to post more on that in the near future, but I think it offers a unique learning interface for tackling complex material, which we hinted at with earlier version of Learnstream, but feels more natural here.

In line with my innovation stuff last week, I think getting out open source code is quite important to move along the field. In this case, I was able to extract the code for spaced repetition from Learnstream Atomic to a Rails gem, Spaceable, which I reused in Rudinium. If this turns out to be a useful technique, it’d be great — and not even that hard — to generalize the application.

What I really wanted to talk about was this awesome story about the game Borderlands. Video games are another tool that operate pretty well without a solving the HFAI-complete theory of how people have fun. Not only that, but they’re incredibly complex code wise. Yegge talks how it wasn’t until an expansion version and multiple bugs discovered by the community that the game gets really good. He also lays out some of the factors of addictiveness and claims that the game developers had, perhaps, just a hint of this.

Lessons here? First, I think we should throw some of those addictiveness factors into a learning application and just see what happens. Maybe the developers at Khan Academy will read the article — their current badge system mostly misses the mark. That leads to the second lesson, which is that once there are a lot of people around you can always extend it based on what it looks like they want. Finally, while keeping it easy to get started, set the bar very high in some parts and throw in complexity: your masses of addicted players — I mean, learners — will figure out something. We do have some high bar educational resources like MIT OpenCourseWare (and people like Scott Young hacking away at it), but what’s missing is an evaluation criterion that doesn’t require Scott’s level of dedication to set up (i.e. taking the exams and self-grading them).

Productivity stuff

  • Researchr works well for PDFs. I can import the highlights from Skim into my wiki and then summarize them. But what about websites? I generally use Evernote, but there are no highlighting features. What I’ve started doing this week is highlighting with Diigo and then importing into Evernote, pre-highlighted. This seems to have pushed me to import a little more. Once I get to importing enough, I’ll have to make sure I can actually make sense of it! (The weekly review is one way.)
  • I’ve talked before about not liking services like Rescue Time very much because they force me to label something good or bad. In the spirit of falsification, I’m trying it out. Indeed, I don’t care too much about the productivity scores I get, but the raw data can be interesting. Apparently I’ve spent 12 hours in Gmail in the last week! How much of that is chatting, reading, writing, doing nothing, etc? Apparently I need to buy the pro version to figure that out (or I can get a hint of it by using an external chat application). But that number alone tells me I need to think about this. Likewise, I have a lot of time in “productive” applications like vim, but how much of that is driving toward a clear goal, and how much is staring blankly at code?
  • An inspiration for lots of passive data collection: Stephen Wolfram’s personal data from over 20 years. Also his interview with Quantified Self.
  • Neal linked me to this post which has a Richard Feynman quote that we’ve been trying to hunt down:

    You have to keep a dozen of your favorite problems constantly present in your mind, although by and large they will lay in a dormant state. Every time you hear or read a new trick or a new result, test it against each of your twelve problems to see whether it helps. Every once in a while there will be a hit, and people will say, “How did he do it? He must be a genius!”

    Actually I do keep a list of research questions, which is currently at eight, but I don’t think I have them constantly present in my mind well enough. To do that, I can a) work on refining them and restating them in various ways b) explicitly organize around them, such as by making Evernote folders and dragging in related material when I come across it. I’m thinking I’ll try to get to twelve research questions plus about eight extracurricular interest questions.

    Despite being from Feynman, it could be a bad thing to do, especially when implemented by someone other than Feynman. One question is whether twelve is the right amount. I’d say it’s enough — many careers have been made on less. It’s certainly not too many to store in memory, and I’d say it’s not too many to test something against each. Another question is whether biasing your focus towards these topics may thwart your observations of interesting things. I’d wager that most observations are made with either not enough focus (“Something about technology. Huh.”) or with too singular a focus (“There must be a way to use this new garbage collection algorithm to improve math education for eight graders!”). But it’d be worth monitoring… if that’s even possible. I also think it’s important that they are worded as questions rather than assertions, so both confirming and falsifying evidence are more likely to be gathered (maybe).

Weekly review through March 11

Innovation in HCI

Given that you’d like to see some change in the world — let’s say the world of software — what is the best way to go about achieving it? Do you try to do it yourself? Add it to an open-source project? Demand it from an existing company? Publish an idea at an academic conference? Just give up and let the world work it out?

My approach has always been to build something new. More specifically, I’ll get very excited about an idea and immediately think about how to implement it. Then I’ll search through what exists and go back and forth between thinking I’ve been beat to it, and thinking that the existing stuff is so terrible I can achieve fame and fortune with just about anything. Sometimes I’ll entertain the idea of adding it on top of an existing platform (we were so excited about Google Wave!). In the end, I’ll build something. Sometimes it fails, sometimes it turns out okay. (None have yet to gain me fame and fortune.)

Now that I’m in an academic field, it’s worth re-examining my approach. The way it’s called around here is “contribution”. As in, “Oh, that system you built is nice. But what’s the contribution?”

The first question is whether innovative system design can be an academic contribution at all. One type of contribution, in the tradition of the natural sciences, is establishing a basic scientific truth. The problem is that when dealing with technology, the truths that may be relevant to us are rapidly changing. For example, “it is impossible to stream a movie over the internet with existing technology” was a truth not long ago. To do research, therefore, either we must find an abstraction that may remain constant, or we must be on top of the changes. One way to do that is, of course, creating those changes ourselves through innovation. (See Pasteur’s Quandrant.)

Still, innovations-as-contributions (both academically and realistically) require two criteria: 1) whether we are able to implement them at all 2) whether they are diffused into society, or otherwise relevant to society (e.g. used only by a few people but with a propagating effect).

The first goes back the questions that I started off with. It’s generally a question of efficiency (and sometimes possibility). For example, to examine an innovation in reference management, I could try to build a feature on top of the Mendeley API or ask them to implement a feature. Another possibility is what’s called a Wizard of Oz experiment, where the system is not actually implemented but in some way artificially produced. But many of my ideas are easily implementable.

Next is the diffusion criterion. But if you could answer this, then you’re ahead of the game. In other words, if you could build a genuinely new system that will be diffused into society, then you already have an interesting contribution: that society wants something that doesn’t exist. In reality, the question is very difficult. People doesn’t necessarily want something more efficient, or prettier, and they aren’t just a standing reserve for any given crowdsourcing application (see also Bennett, Maton & Kervin, 2008). The best heuristic may be that there is a demonstrable problem that people have.

How often should we see innovation happening in, for instance, my HCI department? A lot of work here — good work — is looking at existing systems like Wikipedia, Facebook, Mechanical Turk, Cognitive Tutors. Things that are currently shaping the world. And they are all ones that seem like they’ll be relevant for a while. There are also a lot of people working on innovative systems. Yet for some reason, I feel a sense of resistance to innovation. If it isn’t imagined altogether, it may be more specific to innovations that are targeted at us the students as users. Maybe those should be rare — we are a quite specific subpopulation — but all of my interests have been in that category: learning, productivity, reference management. There is at least one counterexample (in my first year): an event planning app called Happoning. So maybe it’s just me…

One path to diffusion

The particular innovation I’ve been thinking about lately is getting academics to share their favorite references plus some of their knowledge about them (how to interpret them, what makes them important, etc.). The current de facto standard right now is Mendeley. It has its own massive library, groups for sharing citations, and web and desktop tools for organizing and reading papers. I never liked it, and I don’t think I’m alone — only 12% of our department indicated that they use Mendeley.

Our problem is not being better than Mendeley. It’s getting people who don’t find a need for Mendeley.

The original approach I took to sending references information to targeted users is to create groups — basically the same model as Mendeley. What I realized is that people aren’t going to regularly check a fledgling website. Even my first few collaborators haven’t seemed interested in trying it out. For a long time, even Facebook didn’t assume that people would check the site — it sent out lots of emails. Anyway, my goal is not to become Facebook, but I realized that email (and blogging) is the way that people are going to be sharing references for a while to come. So rather than create a new protocol for communication, I’m going to think about this project as a set of features to make it more convenient to share references via email.

As some inspiration, Paul Graham talks about frighteningly ambitious ideas. I would not say that a new reference management tool is frighteningly ambitious, but for me it alludes to a wider goal of disrupting the distribution of academic knowledge. There are a number of interesting points there, but the relevant one is that diffusion within a small group is a good indication of wider diffusion: “If you can just build something that you and your friends genuinely prefer to Google, you’re already about 10% of the way to an IPO, just as Facebook was (though they probably didn’t realize it) when they got all the Harvard undergrads.”

Perfect Health Diet

Now for something completely different. This book is great. It lays out a very well-researched diet suggestion including macronutrient profile, foods to avoid due to toxicity, and micronutrients to include to avoid malnourishment. Here it is in one picture: http://perfecthealthdiet.com/?page_id=8.

The usual argument for paleo diet is “we evolved to eat in the hunter-gatherer style, so we should avoid grains and other manufactured food.” It’s not overwhelmingly convincing on its own. PHD lays out a robust argument for its touted macronutrient profile (65% fat, 20% carb, 15% protein): 1. it’s similar to how our ancestors ate, 2. it’s similar to how other animals eat, 3. it’s similar to breast milk, 4. it’s similar to the human body composition, which is consumed with naturally occurring self-cannibalism. Why not build up a theory from scientific knowledge of chemistry and biology? Or on the results of randomized trials? The medical literature is overwhelming, plus RCTs can only answer very specific questions over limited time periods. This foundation is a more holistic picture. It doesn’t always work, e.g., it can’t answer which foods today have toxins.

Is there a similar argumentation style for education? I’ve claimed before that the point of education is internalizing artificial knowledge, so it’s difficult to study as an analogy to natural phenomena. Perhaps abstracted as information processing? Cognitive load theory, for instance, should be observable in natural human behaviors. What about the use of analogy or the self-explanation effect?

Weekly review through March 4


Hattie & Timperley, 2007 have oodles of sources on feedback, and they organize them nicely. I also have Hattie’s book Visual Learning but haven’t read it yet. They first claim that feedback can answer three questions: Where am I going? How am I going? Where to next? Feedback also comes at several levels: at the task/product (FT), at the process (FP), applied to self-regulation (FR), or toward the self (FS). I also read Butler & Winne, 1995, but it was a little more abstract and hazy — the gist is that a learner needs to apply feedback to improve knowledge, tactics, strategies, and beliefs, but that doesn’t necessarily work due to missing the feedback, having negative affect, filtering through existing beliefs, etc.

Applied to my own life, I’ve been feeling frustrated with my activities because I feel like I’ve plateaued in many areas. What kind of feedback would help? Rate of progress is generally a “How am I going?” question, but as Hattie and Timperley point out, it can be important to think about “Where am I going?” and how that should affect “Where to next?”.

  • I may not be realistic about the extent to which I’m performing poorly. Anything looks pretty flat from a short enough perspective. Some things (Chinese characters, weightlifting) I am tracking well so that I can eventually look at longer trends, others not as much. I should start as soon as possible if I really care about them.
  • Feedback for weightlifting has been difficult. I occasionally get a guy in the gym telling me I’m screwing something up, but their feedback is not very specific. (FS-that-should-be-worded-as-FT: “You’re going to screw up your back!”, useless-FP: “You should go to a lower weight.” Perhaps, but then what?)
  • The web has a lot of advice, but how I do I turn advice into feedback? First, there’s the issue of whether to believe the advice at all. It may just be pretty strong and consistent, so I can believe it as is. Otherwise, it may be something I can test experimentally. Or I may want to get an understanding of the underlying mechanism. OK, so it’s basically just doing science. Here are examples of each:
    • Something I’ve seen everywhere is that you should eat 1 gram of protein per pound bodyweight. So I started tracking protein and found out I was eating a lot less than I thought! Easy test, easy fix. (Although I ordered a food scale to get more accurate measurements when I’m cooking meat.)
    • For various suggestions about form, I’ve tried simple experiments: do it next time at the gym and see whether it works. This isn’t the best way to experiment, but sometimes the result will be obvious. Figuring out the crucial things that are worth serious experiments is something to aspire to.
    • I’m still confused about the best way to eat pre- and post-workout and how those fit in with my diet, so I’m going to try to get a better conceptual model of metabolism.
  • The advice I get from bro scientists at the gym is pretty infrequent, so I mainly rely on assessing myself. I think the previous articles generally assume feedback from an expert, but I’m trying to bootstrap my own expertise in most cases. For weightlifting, again: Whether I lifted the weight is an easy one. Whether I lifted it right is another story. I’ve been asking a friend to video record to get a better picture about how it looks, and I can compare that to images and videos on the web or other people at the gym. The hard part is figuring out how to get from one to the other (which could be a difference of process, or could just be body differences!) I would really like to work on better tracking for this process.
  • “Where to next?” comes up occasionally. Weightlifting is really just about lifting heavier things (if weightlifting is what I care about). Chinese, for now, is about more characters (if I even want to know how to read and write Chinese). Research is, of course, about understanding and synthesizing knowledge, brainstorming innovations, performing experiments, analyzing data, communicating results, and networking in the academic community (if I ever want to get a job). The “where to next?” question also works at different levels. Those, deciding whether to even do these things, are at the self level. But there are decisions at the process level, like, when do I try a different routine? And at the self-regulation level, which I have sprinkled into the previous points.

Short thought

Programming means I get nothing else done. My average time to sleep this week was 3 a.m. No good. That’s all I got for now.