A couple of years ago I wrote a little advice for systems papers writers for the CHI 2011 Facebook group; with the CSCW deadline coming up and CHI closer than you think in the rearview mirror, it seems useful to resurrect here.  It’s imperfect advice, and others will disagree; I’m hoping they’ll put some of their own bits here.

The tl;dr: Tell readers why your work matters, not what you did, and think about what they can take away. Bonus list of crash landings at the end, along with snarky but useful footnotes and pointers to very useful articles.

The long form: Cliff asked me (in 2010) to write a little bit about what makes for a good paper with a significant system component in CHI. Henry Lieberman’s The Tyranny of Evaluation, Saul Greenberg and Bill Buxton’s Usability evaluation considered harmful (some of the time), James Landay’s discussion of the pain of systems work, and CHI’s own guide to successful archive submissions have said a fair amount already, and you should go read those, too.

But, as someone who’s reviewed dozens of papers a year for the last 4 years, I’ll cheerfully weigh in on what I look for as a reviewer, tempered by things I’ve overheard in PC meetings. You’ll have to forgive all the footnotes; I just read a long law review article and they’re all the rage in that discipline. This is also more collected thoughts than careful treatise. So, caveat emptor.

First, your system, itself, is probably the least important, least interesting thing in your paper [0]. A lot of papers read like a story about “what I did on my summer vacation” [1] (literally, often, because of the rhythm of internships and HCI conference deadlines [2]). Here’s my idea. Here’s what I did. Isn’t it cool? And it’s natural to focus on yourself and what you did, because you know it best.

But people didn’t care about your summer vacation when you were talking about it in fifth grade, and they don’t care about it now, unless it matters to the field or to society at large, and unless you did it well [3]. To tell that story, “what I did” is like the skeleton — and like most skeletons, it’s a little creepy if you see it all by itself.

Instead, think of the raw work as the basis for talking about things that other people can learn from. Sometimes, people can learn from “how I did it”. In some scientific traditions, providing enough detail that someone could implement approximately the same system if they wanted to replicate the work is really important [4]. Especially if the mechanics of the implementation are novel, or informative because they illustrate problems or approaches that might have broad applicability, then they become interesting and worth spending precious writing time, paper space, and reader attention on. But if the guts are mostly things a competent senior could do, they’re not the important part.

The real action is on “why”: why what you did matters [5]. Many aspects of this are outlined in the CHI guide to successful papers  and the reviewing guide , both of which you should read every year [6]. These include demonstrating a contribution, originality, validity of the work, and offering benefits to the reader. With design/system papers, there’s such a nice paragraph there that I will just quote it:

“[Reviewers] often criticize authors for conducting studies without adequate theoretical basis, or for not providing enough evidence or sound reasoning for claims. A further concern is lack of justification for design choices and not explaining why certain design features have been included. In summary, you should explain not only what you did, but also why you did it, so that readers (including reviewers) can be convinced that you made appropriate choices. Explaining your choices can also stimulate more research by helping others see alternative approaches.”

One way to think about that little nugget is that people want to understand why this system: why is your system is arguably “right”, or “better”, or “interesting”, or “useful”, in the context of your problems and your contributions. A solid evaluation of the system/design/technique in use through some combination of usability testing, lab studies, field studies, and logitudinal deployment (the right technique(s) should be chosen based on your problems and contributions), showing its potential or actual value, is one way to do that [7].

But it’s only one way. You can use theories and empirical work about specific designs, about individuals’ capabilities and goals and values, about psychology or sociology or economics or S&TS or [insert your favorite discipline here] to show that your problem is important and your system is a reasonable response to that problem. You can also use these, as well as making parallels to other problems and fields, to argue that your system has a greater, general value, and that many researchers can benefit from it. You can argue that a system that successfully does X might improve lives or the world. But make the case [8].

Many of the same techniques can also be used to motivate specific design choices. You can reason about alternate systems, and their choices, and consider alternate choices [9] and why they might be better. Don’t limit yourself to the academic literature, either; there are a lot of non-research systems that work just fine, and comparing to/critiquing/borrowing from/expanding on those designs is working smarter, not harder. You can appeal to your own experience in the past, to iterations you did along the way, to user studies you or other people did to give you insight into the design. But again, as a consumer, I want to know why this system and not that one; the considerations you had and tradeoffs you made are money in my bank as a designer and as a researcher [10].

So, that’s my rant, for now; hopefully it’s useful. Remember this is my perspective, not a universal guide to CHI success (reread the guide to successful submissions!), and that other, really smart people will disagree with me on some points. But I think most reviewers would agree with the following list of crash landings:

• Fail to motivate why the problem is important.
• Don’t tell me why your system is a plausible approach for that problem.
• Do a cursory job of talking about related work that doesn’t help me understand how yours fits in and what you took from it.
• Spend your time on minutae of implementation that doesn’t matter.
• Present your system as though, like Athena, it sprang fully formed from the head of Zeus.
• Avoid showing me what I can learn from your choices, or talking about the general issues other designers might face.
• Perform a bad evaluation, and/or fail to provide other kinds of justification.
• Make me guess what your contributions are, and how one might apply your work in other contexts.
• Write badly, confusingly, disrespectfully of readers’ time.

Don’t crash land. You might not get in anyways; sometimes the work is not ready, sometimes it gets bad reviews, or bad reviewers, or tired or grumpy reviewers who make a mistake. And sometimes you’re just unlucky. But you’ll do better, more useful, and more publishable work by thinking about how to communicate “why” and not just “what”, and thinking harder about the reader and not the writer of the paper. And that’s a good thing.

— footnotes —

[0] Unless you have a truly bad evaluation, in which case that is the least interesting thing in your paper. The general sentiment at the UIST 2009 PC meeting was “I’d rather see a paper with no evaluation than a bad evaluation”.

[1] And let’s not get started on “what was done on the summer vacation provided to me”, passive voice, allegedly dispassionate and detached boring writing style that papers affect and that makes readers die a little inside. Read The Elements of Style, or On Writing Well, and take it to heart. Readers will thank you, and they will probably feel a little better about your paper as well.

[2] Johnathan Grudin, among others, has been advocating for a move toward journal publication over the last few years, to get at deeper, better research. There are practical career values to doing some journal-based work, too; I didn’t get a job out of grad school in part because I didn’t have any journal papers — direct quote from a respected mentor. I’ve also gotten advice from several people that even if your department, or the field, doesn’t care about journal publication, your school or college might. Do you really want your tenure application to go down in flames because of an angry group of civil engineers or economists or historians? So journals seem like a good idea, although please don’t let that stop you from submitting to CHI along the way!

[3] People who have written NSF proposals should have heard “intellectual merit” and “broader impact” when they read that sentence.

[4] Based on CHI reviews and PC meetings I’ve seen and done, as well as the explicit statement in the Guide to Successful Papers about originality, replication is much more likely to succeed if there’s a novelty component as well. Jeff Heer and Michael Bostock’s CHI 2010 replication of some classic information visualization perception studies using Mechanical Turk to solicit participants is one such.

[5] If you can’t clearly articulate this, you may be doing it wrong. Go read Richard Hamming’s You and Your Research immediately. Crotchety, but valuable. Then, as an encore, start working your way through Phil Agre’s Networking on the Network about how to manage the professional side of the career. It’s not perfect, but valuable.

[6] And if you’re not reviewing, shame on you, you freeloader. The average submission gets attention from 5 people, so the authors of every submission should conspire to do at least 5 reviews.

[7] There is an unfortunate perception that to get a systems/design paper through the CHI reviewing process you need an evaluation. Not true. You need a good evaluation. 🙂 Or a really good non-evaluative justification. Both is even better.

[8] Usually in the introduction. You’d really like your readers to want to read your paper, so convincing them it’s important and valuable work should start early.

[9] That is, don’t just list related work, but use it, respectfully, to talk about your system, your choices, and why what you did is novel, interesting, and valuable. And try to avoid the style you sometimes see in papers where the related work goes at the end and is mostly used to talk about limitations with that work that the current paper heroically overcomes. It’s awfully easy for that to sound like “look how dumb they are and how smart I am”, and that’s a big turn-off to reviewers, especially if (as is often the case) they did some of that work.

[10] You should be writing this stuff down as you go along, so you can talk about it later, and keeping every version of the system that matters, even a little bit, for helping people — including yourself — learn more from your work. Bonus points if you tell me about your failures, and what didn’t work. I realize it’s scary to talk about the making of the sausage, and it’s hard to get an epic fail published, but things that didn’t work along the way are useful.

Good luck, fellow travelers. And please, add your own thoughts.