In case you were wondering: no, it’s not that comfy to be an actual area chair, though you might get to enjoy a lot of reading!
Area chairs (AC) are volunteer service positions at academic conferences (or large workshops) that are tasked with overseeing a small part of the peer review process. These positions are typically associated with an area of expertise for the chairperson, thus the name.
Despite the word “chair” in the name, in large academic conferences today there are often a large number of area chairs to share the responsibility of making sure thousands of papers are properly reviewed in time. Some conferences have introduced the role of senior area chairs in recent years, which are usually filled by seasoned researchers, to provide better uniformity in review outcome by aggregating information from area chairs. At the very top, a conference is usually run by a few program chairs and general chairs that are responsible for steering the conference and making sure every aspect of it happens as smoothly as possible (see Jason Eisner’s post for more details about what a program chair does – spoiler alert: it involves great power and great responsibility).
In short, program chairs and general chairs oversee the entire review process via senior area chairs among other conference logistics; senior ACs gather information from ACs, who in turn are responsible for direct interactions with reviewers on individual papers.
To the best of my knowledge, none of the chairpersons actually receives a piece of comfortable furniture for doing what they do.
In short, everything that directly interacts with authors, papers, and reviewers in the review proccess.
If you search for “what does an area chair do”, the first few results will typically contain the following responsibilities, which vary slightly from event to event (sources: WIFS 2017, CVPR 2013, NeurIPS 2018):
Facilitate discussion after author response. Once all initial reviews are finished, conferences typically have a period of “rebuttal” or “author response” so that authors can address reviewer questions and clarify confusions, if any. Once author responses are submitted, the area chair is responsible for engaging reviewers to read the author responses, incorporate them into their final review of the paper, and hopefully converge on a consensus regarding the paper’s acceptance recommendation (no, consensus doesn’t always happen).
Area chairs also occasionally receive confidential feedback from authors on review quality, which they need to incorporate into consideration when moderating discussion.
Write meta-reviews and make acceptance recommendations. Meta-reviews are summaries of highlights (and lowlights) in all reviews regarding a submission. They are useful for senior area chairs to calibrate between AC recommendations, and for the authors to understand the high-level gist of the reviews if they are released.
While area chairs cannot directly determine whether a paper is accepted or not, they do have a large influence with their initial aggregation of reviewer stances. ACs are typically tasked with making a recommendation for each submission, which are then calibrated by SACs before they are presented to the program chairs for a final decision.
In a typical conference, an AC will oversee 10-20 submissions, and most of the work is concentrated within a small window of a few weeks, when they are expected to be highly responsive.
If you are still reading to this point, you are probably like my wide-eyed self a few months back, wondering what life will be like after you’ve earned the degree you have worked years towards, especially what “growing in seniority” in the academic world looks like.
The list of responsibilities give you a rough sense of what is going to happen, and turns out to be quite accurate. However, I find it insufficient in preparing aspiring ACs for how to best do their job, and what to expect in the process, which is usually expected to be picked up on the job.
In the interest of transparency and to save future ACs and SACs the last-minute scramble, I’d like to share some of my personal experience about what things look like in terms of actual work done.
Assign reviewers. ACs usually aren’t involved in this process if SACs are available in the review process, because the reviewing process is double-blind at the AC level (i.e., ACs don’t know who the authors are, and vice versa). For conferneces with paper bidding, reviewer preference is considered jointly with their expertise. Then SACs solve a constrained bipartite matching problem to sort out conflicts of interest, reviewer capacity, etc etc before handing papers with reviewer assignments to the ACs.
Once reviewer assignment has been made, ACs have a job responsibility that is not explicitly covered in the bullet points: sanity-checking reviewer assignment. There is typically a window of a few days when ACs are tasked to ensure each paper has at least one experienced reviewer assigned, and that their general expertise is relevant to what the paper is about. If that were not the case, ACs should notify SACs to adjust the assignment accordingly (usually by exchanging a couple emails). This involves a lot of searching for people’s background and experience, where a quick link to someone’s Google Scholar (or similar) profile can really help.
Review assignment is finalized before the official reviewing window begins for reviewers to read their assigned papers and write up reviews.
Chase late reviews. Once the official review window has passed, there is typically a short window of buffer time for ACs to chase late reviews, and find emergency reviewers to fill in if possible. Chasing late reviews involves sending individual emails to each late reviewer regarding the paper they are assigned, and gentally nudging them to submit the review as soon as possible. Using individual emails is important because some conferences choose to be triple blind (i.e., reviewers don’t know each other’s identity), and keeping different papers in different email threads helps you, the AC, better manage your pool of submissions. There will likely be more late reviews than you would expect, because (in part) life happens and people have unexpected obligations come up. I sent out my “chase” email the day after the official review window, which gives authors the time to respond, and myself the time to act upon their response (or lack thereof).
After AC’s chase email, some reviewers will respond that they will (a) submit late but still before author response, which is great, or (b) not be able to review due to unexpected conflicts, which is not so great, or (c) not respond at all. When the latter two happens, an AC would have to find alternatives – emergency reviewers. Each conference would have collected a list of emergency reviewers, and the AC’s job is to find them to (heroically) fill in when the initial reviewers cannot make it. An AC would need to sift through the list of emergency reviewers, and essentially redo part of the reviewer assignment process but with less time – look for reviewers that are likely qualified and available (many will not be available or responsive, unfortunately).
Once emergency reviewers are identified, ACs need to reach out to these emergency reviewers for their consent to help review. I also shared paper titles so emergency reviewers can judge for themselves whether they would be interested. If the reviewer agrees, ACs would then need to float the name to SACs to add them to the paper and check for conflicts of interst in the process (which ACs cannot check due to the blind process). Once that is done for all of the papers missing reviews, ACs will need to work closely with emergency reviewers to make sure they are able to submit reviews on time. Balancing workload for emergency reviewers is also key given the extremely short time frame.
Facilitate discussion. After the authors had a chance to respond to initial reviews, ACs typically need to find concensus among reviewers regarding their recommendation of the paper. If there weren’t enough time prior to sharing reviews with authors, this is also a great time for ACs to call out reviewers if their evaluation is not clear, or if their criteria are not compatible with reviewer guidelines.
I personally encouraged all of my reviewers to read the author response, and update their final reviews accordingly to reflect the fact that they have read it even if the reviewers are already in agreement. If they are not, an AC can help reviewers elaborate their respective points by asking specific questions in the discussion, and helping refine their reviews to be more objective and provide constructive details that can help authors.
It is not uncommon, however, when reviewers fail to participate in the discussion or update their reviews. In these scenarios, aside from gental reminders in the discussion, an AC would also need to take a closer look at the reviews, author response, and even the paper itself to seek answers for how to objectively evaluate the submission.
Write meta-reviews. If an AC made good use of the discussion period to understand the paper and review opinion, writing the meta-review should be a good opportunity to summarize the high-level takeaways for the SACs to make recommendations to the program chairs. In conferences where meta-reviews are shared with the authors, this is also a good place to give the authors a constructive tl;dr for the final reviews.
Depending on the rules of the conference, the AC might still be able to discuss with reviewers during this period to clarify some points, but really most of the work is already done, and ACs are just writing up a coherent summary. I personally found maintaining a bullet point list of pros and cons for each paper in the reviews and the discussion to be helpful.
Finally, ACs will need to calibrate among all the papers they are in charge of overseeing, and make initial acceptance/rejection recommendations for SACs to further aggregate and calibrate. There can still be a lot of “maybe”s at this stage except for very clear accepts/rejects.
Seeing that the post is already growing a lot longer than I had anticipated, I thought I would end with some short but concrete takeaways for future reviewers, ACs, SACs, and program chairs to consider.
Reviewers. If you have research experience and especially have published, you are highly encouraged to sign up to review somewhat regularly! If you haven’t reviewed much, you are encouraged to seek mentorship (ACL 2021 is taking a great step in this direction).
In the meantime, please try to anticipate potential bandwidth constraints when agreeing to review. Bottomline, please try to be responsive during the period you had agreed to review, so that ACs can at least look for your replacement with sufficient notice.
ACs. If you have been invested about the reviewing process and think you might be eligible, reach out to a senior AC in your area of expertise! You might be surprised with a warm welcome – we are all struggling to keep up with the growing number of submissions these days.
Once you are serving as an AC, please try to be more responsive than if you were a reviewer, as this could really help improve the entire review process. But also keep in mind that you will likely stumble upon unresponsive reviewers or emergency reviewers. Give yourself a lot of lead time and send 2x the email out at once if needed, because response takes time. Work with other ACs in your area might also help in the event of a emergency reviewer conflict of interest – you might end up being able to “exchange” emergency reviewers who are already responsive and willing to review.
SACs. Please consider sharing as much resources as possible ahead of time with ACs, including training material (if any) and emergency reviewer info. This could help reduce your level of anxiety during the review process caused by unnecessary communication.
My NAACL SACs did a great job at providing what is available, but I do wish there were a bit more training one could complete in prior so I can start the job more prepared.
Program chairs. This is more of a comment about the reviewing software/features to consider, rather than about the chairs themselves, who I believe are doing a great job within their capacity. While the overall process was smooth for me, I find myself struggling from time to time with small issues that prohibited a potentially even smoother experience. I have personally experienced this in two places:
The first involves transparency regarding withdrawn papers. The information about desk rejects and withdrawals during author response were either not shared until a few email exchanges later, or not available in a prominent interface to ACs/reviewers. A “simple” status change in the review system can easily result in hundreds of emails and the delay/confusion associated with them, given the size of our conferences nowadays.
The second involves reviewer interaction and reviewer conflicts of interest. Never having been on the other end of the “chase” email, I had never anticipated having to send them one by one from my personal mailbox, filling paper information into the email template I drafted for myself. I was also surprised to find out that one of the reviewers never agreed to review in the first place. When it comes to emergency reviewers, there was no easy way to check for COI other than floating names to SACs, who are authorized to add them or tell us if there had been a COI. While the need for a centralized management of sensitive information is very much understandable, it is at these times I couldn’t help but think how much time a semi-automated system could save everyone.
Overall, I am excited to have had the opportunity to serve as an area chair, and to be able to share my experience that hopefully helps others. I am also encouraged that the field is actively taking measures to improve the review process (shout out to ACLRollingReview). Can’t wait to see how we can collectively improve the reviewing/publishing process in the next few months/years!