Thoughts on #recsys2016 people recommendation tutorial

Rather than tweetstorm, one post with some reactions to the people recommendation tutorial at #recsys2016 by Ido Guy (Yahoo Research, Israel) and Luiz Pizzato (Commonwealth Bank of Australia, Australia), primarily Luiz’ part.

Not sure people recommendation has to be symmetrical; it would be interesting to ponder use cases where that’s not true (Twitter is the one that comes first to mind).

The point that success often gets measured outside the system is a good one — the example of a dating site being successful when people stop using it because they matched was cute. Victoria Sosik, Steve Ibara, and Lindsay Reynolds thought about that in persuasive systems, and the Suhonen et al. Kassi system in barter and social exchange systems.

Random side thought: wonder about designing not just a dating site but a relationship one, where you might use the site to help continue and develop the relationship long term. Doug Zytko was thinking about this a while ago.

Claim that successful people recommendations are ones that lead to interaction is a little overstated maybe. For instance, one potential benefit of “unsuccessful” recs that don’t lead to interaction is learning more about the space of possible people — what is this social circle or company or population of mates _like_?

Point that unsuccessful people recommendations can have psychological effects was nice, as was the idea that people might be more inclined to be makers or receivers of people recommendations, and that varies by context…

The point that people might become less picky over time if early attempts on dates, jobs, etc. don’t pay off (or more picky if you have much success) was interesting, but is that algorithm-actionable? Or is that more about how far people are willing to go down a list of recommendations?

Fraud concern feels a little overstated, even with the legitimate threats of fudgers, liars, scammers… Reminds me a little of the obligatory preventing collusion section in crowdsourcing papers.

Why I’m rotating at NSF

tl/dr: Being a temporary program officer at NSF comes with real job and life tradeoffs; for me, the job tradeoffs around learning, service, and impact felt good, and the life timing turned out to be surprisingly good. So, I took a chance, and I’ll see some of you at NSF as panelists and others at conferences wearing my NSF hat over the next couple of years.

More details:

Last month I started a new academic adventure, as a rotating (temporary) program director (PD) at the National Science Foundation (NSF) [1], in a program called Cyber-Human Systems (CHS) [2]. Some people might wonder why and how this came to be, either out of curiosity or because they, too, might consider it someday [3].

First, some background on being a rotator: NSF regularly brings in outside folks for fresh ideas, energy, and connections to emerging/priority intellectual communities and fields. These assignments typically run about two years [4], occasionally longer, and you apparently do most everything permanent folks do: run panels, make funding recommendations, administer existing grants, collaborate with other programs and other funding agencies, and presumably things I’m still not aware of.

Now, the “why”. One answer is that I’ve thought about doing this for a long time. I found panels and proposal reviewing fun, and former program officers suggested that I might be good at it [5], so it’s been floating around in my head. I put together my own experiences as a panelist with readings of NSF’s own materials [3] and the testimonials of former officers [6], then had a number of conversations with former PDs, folks in the greater CHI/CSCW community, and people at Cornell. This all added up to me seeing real benefits (with some tradeoffs) around learning, service, and impact as a program officer.

I was pretty sure I would learn a lot, both about the broader field and about NSF itself. There’s a lot of territory in CHI and CSCW I don’t see so much, and I figured this would put me on a collision course with new spaces and give me a great bigger-picture view. Further, having more intimate knowledge of how the sausage gets made [7] at NSF was appealing both for its own sake and as a practical tool that, along with the perspective, would benefit both me and Cornell down the road. The downside risk for me is that I’m pretty broad already, and in some worlds the job responsibilities might encourage an awkward breadth-depth tradeoff.

Service. I like helping others — reviewing, commenting, advising, organizing, providing opportunity — and this is clearly a venue for that. Good reviews and process sometimes help PIs push on their ideas [8]; I’ll have plenty of chances to interact with PIs directly [9]; choosing and mentoring junior panelists can help them grow in their careers [10]. All of this has real import for a lot of people. The downside risk here is that the responsibilities trade off with doing your own research, and pretty much everyone said that productivity goes down during (and if you’re not careful, after) the rotation. NSF does give PDs up to 50 days a year of Individual Research and Development (IR/D) time to work on your own stuff, but that’s still much less than I was spending over the last several years.

Impact. I think I have the chance to have real impact, both in the small around particular proposal decisions and in the medium about encouraging kinds of work that I think are important. In the small, there are usually more awesome proposals to fund than dollars to fund them; panel input is taken quite seriously but program directors still make plenty of decisions about which of the good set to recommend for funding [11]. In the medium, you get to interact with folks at NSF and other agencies and try to convince them to allocate money in directions you think are important [12]. This might require more people skills than I have, but we’ll see. On the downside, as described above your direct research impact is likely to go down for a while. A few people also suggested that there might be value instead of rotating in waiting and taking a more senior temporary position (perhaps as a division director rather than a program director, or in agencies where individual program directors have more individual power).

In the end, I think the benefits beat the costs for me in the abstract, which brings me to the concrete “how it happened”. I’ve been told that there is an NSF policy that you need to be at least six years research-active past the PhD so although I had pondered it, it didn’t become seriously plausible until about 2013.

In fall of 2014 Kevin Crowston‘s rotation [13] was scheduled to end, and someone asked if I might be interested in trying out for the team. At the time I said no; Lindsay and I had just bought a house, we were getting married in a couple of months, and I was seriously thinking about what to do for a sabbatical after not planning one before reaching tenure in 2013 [14].

There are lots of other reasons why someone would be interested-but-not-willing to do the job at any given time. People I’ve discussed this with mention a number of other good reasons: kids and schools; geographical preferences and spousal prospects; having a lot of students, collaborators, or projects; timing around promotions or lack of support at the home institution [15]; and general risk aversion or different weighings of the values and risks I discussed earlier all sound like good reasons to try it out later, or never.

But in July 2015 at the CSST summer institute [16], I heard that they hadn’t yet found someone to replace Kevin. When I ran the decision process again a few new things bubbled up that made it seem much more plausible.

On the personal side, the new home argument didn’t seem as critical after having lived there for a bit. Talking about the sabbatical had gotten Lindsay excited about trying a new place [17] and her job is portable, making that an upcheck. We also wondered if a longish move now would be better than, say, in 5-10-15 years when our hypothetical kids were in school [18].

On the job side, Cornell Information Science has been growing, meaning me leaving for a while would be less heavy of a burden for the department [19]. A couple of students had recently graduated and most of the others were pretty far along, so on balance I didn’t feel like I would be leaving them in the lurch [20]. I had a pretty positive outlook on the cost-benefit tradeoffs described above. Finally, I was just “called” to it [21].

And there you have it. There was an interview and administrative process that are vaguely interesting, and the finances also are mysterious-but-possibly-beneficial [22], but I still don’t fully understand them so I’m going to wait for a while to write about them — and this post is quite long enough. So I’m done, except to remind you that if you’re interested in serving on a panel — or talking about rotating — at some point, let me know.

#30#

[1] In this, and in all future posts, the views represented are entirely my own, not those of NSF itself nor my NSF overlords. For instance, when I say “my NSF overlords”, I’m pretty sure that’s not how they’d put it. Pretty… sure.

[2] I’m still getting used to NSF structure myself, so, the full story: NSF has directorates that oversee major scientific and administrative responsibilities at the Foundation; the directorate CHS is in is called Computer & Information Science & Engineering (CISE). Inside of directorates are divisions; the division CHS is in is called Information and Intelligent Systems. So, CHS -> IIS -> CISE -> NSF.

[3] NSF has a part of the website devoted to info about being a rotator.

[4] I won’t talk much about the logistics of how it works because there are different paths; my own is a program called the Intergovernmental Personnel Act (IPA), in which you’re technically still employed by your home institution and you return to it after you leave.

[5] Some general criteria former rotators have mentioned, if you want to run a self-test: scientifically good and well-connected to one’s research community (i.e., some street cred); open to but also willing to critically evaluate many ideas and methods (i.e., no axes to grind); putting in real effort at the reviewing task and executing it with competence and timeliness (i.e., you’re reliable); a proven record of community service and working reasonably well with others (i.e., you’re not an asshole).

[6] See, for instance, stories from Doug Fisher and Michelle Elekonich.

[7] Mmm, sausage.

[8] It doesn’t always feel that way when the decline letter comes around. I’ll be curious how it feels to be on the other side; former PD Wayne Lutters described real sadness around declines of worthy proposals.

[9] Here, I’ll need to be careful to keep boundaries, both for fairness and for sanity reasons. One boundary is around investing too much time giving advice about proposals; I think I will find this fun, but that makes it in turn a bit dangerous. Another is around managing cases when people who are angry at reviews, reviewing, and/or me. I’ve been told this is not so common, but that it does happen.

[10] Hopefully, I’ll be able to have a positive impact here around diversity of demographics, perspectives, institutions, etc. And if you want to serve on a panel sometime, let me know; doing and seeing proposal reviews can be really helpful in your own proposal writing — and is also great service and a chance for impact.

[11] So, going back to what makes for an effective rotator: although you shouldn’t have an axe to grind, I’ve been told that you should have some level of vision, and be open to opportunities to encourage it.

[12] Note “recommend”: program directors make recommendations, usually in consultation with other program directors in their programs, about which proposals to fund. Division directors actually have to approve the recommendations, and the final award is actually made by a different part of the Foundation. So when you get that mail about being recommended for funding, remember that it’s “quite probably” but not “slam dunk”.

[13] Who replaced Sue Fussell, who replaced David McDonald, who replaced Wayne Lutters… it’s somewhat Biblical in that way. I’m told that a common tradition is to be called “the new X” where X = you minus 1. So, I’m “the new Kevin Crowston“, I suppose.

[14] I really wasn’t counting my tenure chickens, and there was a lot of luck along the way.

[15] I am really grateful to Cornell, in particular to Jon Kleinberg and now Thorsten Joachims in IS chair and Greg Morrisett in Computing and Information Science Dean roles, for having my back on this.

[16] Nerd camp for people who think that both the social and technical aspects of systems are important to consider when doing either research or development; see the CSST website.

[17] DC is not California, which was the original plan, but so far she still seems excited and happy.

[18] Much less hypothetical now that Gracie has arrived. The timing was pretty funny: I interviewed September 10, got a tentative expression of interest from NSF on the 17th, Lindsay peed on the stick on the 21st, and the CHI deadline was the 25th. So, that was two eventful weeks.

[19] It still wasn’t a light decision on this front for me. Departmental service is awkward as a program officer both because you’re not at your university (with the logistical and interactional issues that come with that) and because you can’t use your NSF-allotted research time for service (i.e., you do it on evenings and weekends). Many academics are no stranger to either of these, but Jon pointed out that leaves really are leaves, and that there’s value in fully committing to new things.

[20] I hadn’t been bringing in students the last couple of years because of a funding dry spell. Conspiracy theorists might speculate that this was an NSF plot, but that’s wrong: I just didn’t have the right proposal sauce for a bit.

[21] Jon described a similar feeling about the Networks, Crowds, and Markets textbook he co-wrote with David Easley a few years back, which I think is part of what led him to support this adventure.

[22] At a high level, your salary is annualized (i.e., I get 3 months of summer salary) and you get a good amount of financial support for expenses for research travel, including to your home institution while you’re there, as well as for moving to/residing in the DC area. It’s not bad.