tl/dr: Treat NSF reports as a required structural opportunity to celebrate, reflect, and plan. Give program officers (just) enough info to understand, share, and think about the cool outcomes and real impacts of the projects.
More detail: My goal with this post is to help you get more personal value out of writing annual NSF reports and also to make them more useful to NSF. I am writing this in my personal role as a faculty member who happens to have experiences as an NSF program officer, not in any official capacity; that said, I’ll talk about my perspective on this both as a PI and based on my experience as a program officer. [A]
Let’s start with the PI side of this, about developing a positive attitude toward report-writing and what you can get out of it. Reports have real value to NSF [B], but the value often isn’t apparent to PIs themselves. My own experience for my first few reports was a little negative. Poking around on the web for report-writing advice turned up phrases like “grit your teeth” and “those darn annual reports”, and they weren’t very helpful. I apparently wrote ok reports and haven’t had one returned, but early on I wasn’t sure why I was doing it except it had to be done.
Then, around year three, I started thinking about the reports as a chance to celebrate, reflect, and plan. It felt good to talk about people I worked with, mentored, and taught, and the knowledge they gained and discovered. It was cool to see how my thinking evolved over the course of the project given circumstances and people, to step outside of the activity of research and do a little bit of meta-level thinking about it [D], and to consider where the work was going and what it meant for the field.
A number of folks have come to a similar framing about proposal writing as a chance to step back and think about what’s important; I encourage you to do that for reporting too [E] — treat reports as a required structural opportunity to celebrate, reflect, and plan. These — especially the first two — are activities that I don’t spend enough time on as a faculty member.
I’m not going to talk much about the requirements or individual sections of the report, as plenty of documents do this [F]. In particular, the Community for Advancing Discovery Research in Education has a nice description of the requirements and most of what I would say would be redundant.
Instead, I’m going to switch gears and talk about my personal experience reading the reports as a program officer, and what made reports more useful and satisfying to consume [G]. Frankly, my early experiences reading annual reports were similar to writing them as a PI: guidance and rationale were minimal [H]. Then an experienced program officer and deputy division director pointed out that beyond the general NSF rationales, reports are the main structural opportunity for program officers and PIs to communicate about awards in progress.
As with the celebrate-reflect-plan framing on the PI side, communicate-engage on the program officer side made reviewing reports a lot more rewarding. It was nice to be able to drop people positive comments on their projects, including occasionally sharing ideas the reports sparked, and it definitely helped me understand areas that weren’t in my wheelhouse [I].
This works best when the report does a good job in the Accomplishments section of reminding me about the key goals of the project and reporting period. Then a thoughtful-but-brief summary of activities is helpful to know how you’re attacking the problem; rambling descriptions are less useful. The most useful reports say a little more about the interesting outcomes and how they contribute to the field. Emphasizing findings is valuable because these reports are a main way program officers stay up on a broad range of projects and fields; we can’t attend/read all the conferences and journals our PIs engage with [J]. Good reports also given useful highlights about the education, outreach, and broader impacts aspects of the project.
I also checked the products and participants sections. NSF wants the products to be correctly uploaded — and papers to acknowledge support — so they can be tracked and associated with the awards. In particular, publications properly entered will appear along with your award abstract in the NSF award database, and that’s useful for us, you, and future folks looking at awards. I often saw issues in the participants section, with PIs failing to list the folks contributed to the project and describe their contributions (see question 4 in the Division of Environmental Biology’s blog post on the topic for useful thoughts on this); this often caused me to send reports back for revisions.
The impacts section is another place I often saw problems. I think this is part because it’s hard to articulate concrete impacts especially early in a project’s lifespan, in part because impact tends to be cumulative in a way annual (or “final”, which is really just “the last annual report” [K]) reports aren’t, and in part because we often don’t spend enough time thinking about the impact of our work beyond the papers. Too many reports default to the same generic, hopeful language proposals often use about potential impacts — in the worst case, cutting and pasting from the proposal. Generalities are not useful, and as a program officer I preferred a report say “nothing to report” on aspects of the impact section rather than make stuff up, or just repeat findings from the accomplishments section (another common approach).
Instead, compelling impact sections give specific descriptions of, evidence for, and/or concrete plans to increase the impact of the project and the underlying research. Are other people reacting to the work, in the main discipline or others, as shown through citations, awards, invited talks, syllabus use, new collaborations, or other concrete evidence that they are thinking about the work? Are students getting valuable experiences and outcomes from project activities, both as research participants and students in courses? Are educational, dataset, source code, implementation, infrastructure, and other materials released to the public, documented, maintained, evangelized, and used by others? Are there concrete possibilities for tech transfer or actual impacts on society beyond “this might be useful, someday”? And for any or all of these, does it make sense to plan activities to increase the chances of having these kinds of impacts? Going back to the lead for this post, report writing should have some benefits for you — and taking a chance to think about how to increase the impact of your work is one of those [L].
And that’s where I think I’ll leave it. As a reminder, this is my own thinking about reports from both the PI and program officer side, and not official NSF policy or prescription, but hopefully it’s useful in helping you both in the writing of your reports and thinking about your work.
#30#
[A] And, as always when I mention NSF, this is my own thinking and does not represent in any official way the opinions of NSF.
[B] NSF offers lots of good reasons to do reporting from an NSF perspective, e.g., accountability for the funded PIs, as well as tracking research and educational impacts and specific outcomes. These are good things to do. Not completing reports in a timely manner also impacts one’s ability to get future funding. That said, these talk mostly about why annual reports are good for NSF, not for you.
[D] For what it’s worth, “go meta” is my number one piece of generalizable advice about being an academic. Don’t just read the paper or go to the talk or listen to the lecture; think about the genre, and what works and doesn’t work for you, and why, and use that going forward. Don’t just review the paper/proposal or do the program committee/review panel; use it as a chance to think about quality science and how people think and talk about it. Then use these meta-insights to be a better reader, writer, teacher, reviewer, community member.
[E] Not everyone buys in; I remember advocating for this at a faculty meeting and being called “Panglossian”. Perhaps true, but it still helps me both feel better about report writing and write better reports.
[F] These include official guidance on NSF’s take on annual reporting (as of 2016 but still current as I write in early 2019), including special instructions for writing reports for conferences/workshops/doctoral consortia and the like, and more info on the mechanics of process and using research.gov to do the reporting.
[G] There are a couple of documents from NSF itself that also have somewhat more detailed thoughts on good report writing, including one from the Brain and Cognitive Sciences division and another from the Division of Environmental Biology. Some of the things I say in this document are based in part on these, along with conversations with other program officers, largely in the Division of Information and Intelligent Systems.
[H] That said, NSF has pretty good high-level training for program officers and a good community of practice that includes both other program officers and especially deputy division directors, the unsung heroes of NSF management who absorb an enormous amount of both corner cases and institutional memory. But it’s got many of the same apprenticeship model characteristics that doing a PhD (or really, being a faculty member) has.
[I] Program officers cover a lot of territory, not all of which is their specific expertise. Further, program officers (especially permanent ones) sometimes wind up adopting awards pretty far from their own areas, for example, when a rotating program officer in charge of certain topics leaves.
[J] Interesting outcomes are also fun to share with other program officers and NSF’s outreach people.
[K] A related issue is that for NSF, a “final report” is not cumulative; it’s just a final “annual report”, and should only cover the last year of activity. This confuses many PIs, and I found I had to return some number of “final” reports for this.
[L] Thinking about providing evidence of impact was also important in my post on writing research statements, so that might be worth a read (and contains pointers to other notions of impact and people who’ve spoken about it as well, including Elizabeth Churchill’s thoughts and Judy Olson’s Athena Award talk).