Long distance usability testing for arXiv
The arXiv development team has been working on a new interface for its volunteer moderators, in order to make their work easier and to decrease the workload for arXiv administrators. When interface development reached a point where we needed to test the result with real live users, we were faced with the interesting challenge of conducting the tests at a distance, as arXiv moderators are distributed all over the world. Thanks to the availability of web conferencing tools, this turned out to be a practical option.
Before we got started, we sought out the advice of Gaby Castro Gessner and Nick Cappadona on conducting usability tests remotely. Their advice included the following suggestions:
- Conduct a technology test meeting to make sure testers have the conferencing application installed with working audio, know how to use the application, and in particular, know how to share their screen.
- Send tasks to testers ahead of time via email, particularly for testers for whom English is not their first language, as the web conference chat function can be cumbersome to use when a test is in progress.
- Offer testers a “lifeline” – a phone number to call or other backup plan in case something goes wrong.
- Send testers informed consent information in advance of the test.
How we did it
I was surprised at how much preparation was required to pull this off, including a lot of scheduling and drafting of correspondence:
- Develop test, test and revise it
- Select dates for technology tests and usability tests
- Reserve meeting room for usability tests
- Invite moderators to sign up for testing slots (Doodle poll)
- Select testers and thank those who weren’t selected
- Write and distribute technology requirements and zoom set-up instructions to testers
- Schedule and set up zoom meetings for tech tests, conduct tests
- Schedule and set up zoom meetings for usability tests
- Recruit arXiv staff to take notes
- Draft and share informed consent information with testers
- Conduct usability tests
- Debrief after tests and clean up notes
- Produce a usable summary for follow up
With the help of Jim Entwood, arXiv Operations Manager, we designed a task-based test that required testers to try to complete the most common and important actions in the interface. Because the work of moderators is highly specialized and has its own distinct conventions and vocabulary, we needed to test the test with someone who “speaks” arXiv. Rebecca Goldweber, assistant arXiv administrator, ably helped us test the test and refine questions and tasks. Chloe McLaren took copious notes. And in the end, we conducted six usability tests, resulting in some very useful and specific feedback for improving the moderator interface.
Advice we’d share
The single most important thing is probably to allow plenty of time to develop the test, recruit testers, and make all the arrangements. From start to finish, the process took about one month. That might sound slow, but developing and testing the test takes time, as does writing invitations and explanatory emails, and corresponding with, recruiting and scheduling testers. I’m sure it will be faster next time, now that we have some experience as well as email text that we can reuse. We look forward to doing more of this as arXiv makes more user-facing changes and enhancements.
Thanks to the arXiv development team and DSPS UX staff for their work on the new moderation interface: Brandon Barker, Brian Caruso, Martin Lessmeister, and Melissa Wallace. And thanks to the arXiv moderators who volunteered and participated in the tests!