A Gentle Introduction to Matching Algorithms in Paper Matching Systems
In academia, publications to journals and conferences typically involve a reviewing process, where the journal editors/conference chairs would match submitted papers to reviewers that have expertise in the material presented in the paper.
The simplest examples of such systems can be described via market clearance matchings. Suppose that we want to match three papers A, B, C to Reviewers X, Y, Z. Each reviewer can bid between 1-5 for each paper, and they gave the following bid:
Then the algorithm for market clearance matching would return the pairs (X, A), (Y, C), (Z, B), where we have set the price of papers to be (A, B, C)=(0, 1, 0). As discussed in class, such matching has the nice property of maximizing the total welfare (i.e. the total score obtained by all reviewers), so it gives the best choice for each reviewer in some sense.
In practice, the paper matching systems are significantly more involved than this simplified example, in the following ways:
- Scoring of each reviewer-paper pair: The score of matching a reviewer to a paper depends not only on the reviewer’s preference (subjective metric) but also on whether the reviewer has expertise toward the topic of the paper (objective metric).
- Support of multiple matching: In reality, each reviewer would be typically matched to multiple papers, while each paper is also reviewed by multiple reviewers. Thus, we will need an extended version of the matching algorithm that supports such extended notions of matching.
- Scalability of the algorithm: Recent conferences, specifically in active areas like machine learning, has a large number of submitted papers easily exceeding a few thousand. This prompts the search for matching algorithms that are efficient.
A recent matching system that has been adapted to many large conferences in computer science is the Toronto Paper Matching System, designed in 2013. It can be illustrated in the following diagram, given in the paper [1]:
The workflow here addresses the above difficulties: There is a complicated scoring system that involves both subjective and objective scoring, and it is followed by a matching algorithm that actually addresses multiple matching in an efficient manner. The readers are recommended to read the paper if they would like to know more about the details involved in such an algorithm.
[1] L. Charlin, R. S. Zemel. The Toronto paper matching system: an automated paper-reviewer assignment system. In International Conference on Machine Learning (ICML) 2013, Workshop on Peer Reviewing and Publishing Models. http://www.cs.toronto.edu/~lcharlin/papers/tpms.pdf