What does an area chair actually do, anyway?

In case you were wondering: no, it’s not that comfy to be an actual area chair, though you might get to enjoy a lot of reading!

What is an area chair?

Area chairs (AC) are volunteer service positions at academic conferences (or large workshops) that are tasked with overseeing a small part of the peer review process. These positions are typically associated with an area of expertise for the chairperson, thus the name.

Despite the word “chair” in the name, in large academic conferences today there are often a large number of area chairs to share the responsibility of making sure thousands of papers are properly reviewed in time. Some conferences have introduced the role of senior area chairs in recent years, which are usually filled by seasoned researchers, to provide better uniformity in review outcome by aggregating information from area chairs. At the very top, a conference is usually run by a few program chairs and general chairs that are responsible for steering the conference and making sure every aspect of it happens as smoothly as possible (see Jason Eisner’s post for more details about what a program chair does – spoiler alert: it involves great power and great responsibility).

In short, program chairs and general chairs oversee the entire review process via senior area chairs among other conference logistics; senior ACs gather information from ACs, who in turn are responsible for direct interactions with reviewers on individual papers.

To the best of my knowledge, none of the chairpersons actually receives a piece of comfortable furniture for doing what they do.

What does an area chair do?

In short, everything that directly interacts with authors, papers, and reviewers in the review proccess.

If you search for “what does an area chair do”, the first few results will typically contain the following responsibilities, which vary slightly from event to event (sources: WIFS 2017, CVPR 2013, NeurIPS 2018):

• Promote your conference/area/workshop. This is typically true for smaller/newer events like workshops. Many larger conferences today have dedicated publicity/social media volunteers to help with this.
• Recruit reviewers. A lot of this work has been shifted to senior ACs when the SAC role exists, who are also responsible for recruiting ACs.
• Assign reviewers to papers. This is usually done with the help of automated assignment software that match reviewers to papers that are suitable for their review interests and past experience (usually evidenced by their published work). For NAACL 2021, the conference I was an area chair for for the first time, this is entirely done at the SAC level. The review process is doubleblind at the AC level.
• Chase late reviews. Life happens and people forget. Sometimes assgined reviewers are not able to submit their reviews on time – it is the AC’s job to reach out and remind them to submit reviews. If that fails, conferences usually have a reserve pool of “emergency reviewers” they can contact for a quick-turnaround review.

• Facilitate discussion after author response. Once all initial reviews are finished, conferences typically have a period of “rebuttal” or “author response” so that authors can address reviewer questions and clarify confusions, if any. Once author responses are submitted, the area chair is responsible for engaging reviewers to read the author responses, incorporate them into their final review of the paper, and hopefully converge on a consensus regarding the paper’s acceptance recommendation (no, consensus doesn’t always happen).

  Area chairs also occasionally receive confidential feedback from authors on review quality, which they need to incorporate into consideration when moderating discussion.

• Write meta-reviews and make acceptance recommendations. Meta-reviews are summaries of highlights (and lowlights) in all reviews regarding a submission. They are useful for senior area chairs to calibrate between AC recommendations, and for the authors to understand the high-level gist of the reviews if they are released.

  While area chairs cannot directly determine whether a paper is accepted or not, they do have a large influence with their initial aggregation of reviewer stances. ACs are typically tasked with making a recommendation for each submission, which are then calibrated by SACs before they are presented to the program chairs for a final decision.

In a typical conference, an AC will oversee 10-20 submissions, and most of the work is concentrated within a small window of a few weeks, when they are expected to be highly responsive.

What does an area chair actually do?

If you are still reading to this point, you are probably like my wide-eyed self a few months back, wondering what life will be like after you’ve earned the degree you have worked years towards, especially what “growing in seniority” in the academic world looks like.

The list of responsibilities give you a rough sense of what is going to happen, and turns out to be quite accurate. However, I find it insufficient in preparing aspiring ACs for how to best do their job, and what to expect in the process, which is usually expected to be picked up on the job.

In the interest of transparency and to save future ACs and SACs the last-minute scramble, I’d like to share some of my personal experience about what things look like in terms of actual work done.

• Promote the conference. I didn’t do much of this as an AC for NAACL, which has its dedicated publicity chairs (which I happen to be one of for NAACL, but that’s a topic for another day). If you’re promoting a new-ish workshop, expect to ask around for access to post on large mailing lists (e.g., the one for ACL members and others for similar professional organizations), drafting mass emails, and sending them out. Social media is also helpful, but you would want to seek help from existing outlets that have a good number of followers.
• Recruite reviewers. I didn’t have to do this either, and I understand that this falls largely on the SACs (who also needs to recruit ACs). This usually involve the aforementioned large mailing lists with a survey form to gather initial reviewer information.

• Assign reviewers. ACs usually aren’t involved in this process if SACs are available in the review process, because the reviewing process is double-blind at the AC level (i.e., ACs don’t know who the authors are, and vice versa). For conferneces with paper bidding, reviewer preference is considered jointly with their expertise. Then SACs solve a constrained bipartite matching problem to sort out conflicts of interest, reviewer capacity, etc etc before handing papers with reviewer assignments to the ACs.

  Once reviewer assignment has been made, ACs have a job responsibility that is not explicitly covered in the bullet points: sanity-checking reviewer assignment. There is typically a window of a few days when ACs are tasked to ensure each paper has at least one experienced reviewer assigned, and that their general expertise is relevant to what the paper is about. If that were not the case, ACs should notify SACs to adjust the assignment accordingly (usually by exchanging a couple emails). This involves a lot of searching for people’s background and experience, where a quick link to someone’s Google Scholar (or similar) profile can really help.

  Review assignment is finalized before the official reviewing window begins for reviewers to read their assigned papers and write up reviews.

• Chase late reviews. Once the official review window has passed, there is typically a short window of buffer time for ACs to chase late reviews, and find emergency reviewers to fill in if possible. Chasing late reviews involves sending individual emails to each late reviewer regarding the paper they are assigned, and gentally nudging them to submit the review as soon as possible. Using individual emails is important because some conferences choose to be triple blind (i.e., reviewers don’t know each other’s identity), and keeping different papers in different email threads helps you, the AC, better manage your pool of submissions. There will likely be more late reviews than you would expect, because (in part) life happens and people have unexpected obligations come up. I sent out my “chase” email the day after the official review window, which gives authors the time to respond, and myself the time to act upon their response (or lack thereof).

  After AC’s chase email, some reviewers will respond that they will (a) submit late but still before author response, which is great, or (b) not be able to review due to unexpected conflicts, which is not so great, or (c) not respond at all. When the latter two happens, an AC would have to find alternatives – emergency reviewers. Each conference would have collected a list of emergency reviewers, and the AC’s job is to find them to (heroically) fill in when the initial reviewers cannot make it. An AC would need to sift through the list of emergency reviewers, and essentially redo part of the reviewer assignment process but with less time – look for reviewers that are likely qualified and available (many will not be available or responsive, unfortunately).

  Once emergency reviewers are identified, ACs need to reach out to these emergency reviewers for their consent to help review. I also shared paper titles so emergency reviewers can judge for themselves whether they would be interested. If the reviewer agrees, ACs would then need to float the name to SACs to add them to the paper and check for conflicts of interst in the process (which ACs cannot check due to the blind process). Once that is done for all of the papers missing reviews, ACs will need to work closely with emergency reviewers to make sure they are able to submit reviews on time. Balancing workload for emergency reviewers is also key given the extremely short time frame.

• Facilitate discussion. After the authors had a chance to respond to initial reviews, ACs typically need to find concensus among reviewers regarding their recommendation of the paper. If there weren’t enough time prior to sharing reviews with authors, this is also a great time for ACs to call out reviewers if their evaluation is not clear, or if their criteria are not compatible with reviewer guidelines.

  I personally encouraged all of my reviewers to read the author response, and update their final reviews accordingly to reflect the fact that they have read it even if the reviewers are already in agreement. If they are not, an AC can help reviewers elaborate their respective points by asking specific questions in the discussion, and helping refine their reviews to be more objective and provide constructive details that can help authors.

  It is not uncommon, however, when reviewers fail to participate in the discussion or update their reviews. In these scenarios, aside from gental reminders in the discussion, an AC would also need to take a closer look at the reviews, author response, and even the paper itself to seek answers for how to objectively evaluate the submission.

• Write meta-reviews. If an AC made good use of the discussion period to understand the paper and review opinion, writing the meta-review should be a good opportunity to summarize the high-level takeaways for the SACs to make recommendations to the program chairs. In conferences where meta-reviews are shared with the authors, this is also a good place to give the authors a constructive tl;dr for the final reviews.

  Depending on the rules of the conference, the AC might still be able to discuss with reviewers during this period to clarify some points, but really most of the work is already done, and ACs are just writing up a coherent summary. I personally found maintaining a bullet point list of pros and cons for each paper in the reviews and the discussion to be helpful.

  Finally, ACs will need to calibrate among all the papers they are in charge of overseeing, and make initial acceptance/rejection recommendations for SACs to further aggregate and calibrate. There can still be a lot of “maybe”s at this stage except for very clear accepts/rejects.

Takeaways

Seeing that the post is already growing a lot longer than I had anticipated, I thought I would end with some short but concrete takeaways for future reviewers, ACs, SACs, and program chairs to consider.

• Reviewers. If you have research experience and especially have published, you are highly encouraged to sign up to review somewhat regularly! If you haven’t reviewed much, you are encouraged to seek mentorship (ACL 2021 is taking a great step in this direction).

  In the meantime, please try to anticipate potential bandwidth constraints when agreeing to review. Bottomline, please try to be responsive during the period you had agreed to review, so that ACs can at least look for your replacement with sufficient notice.

• ACs. If you have been invested about the reviewing process and think you might be eligible, reach out to a senior AC in your area of expertise! You might be surprised with a warm welcome – we are all struggling to keep up with the growing number of submissions these days.

  Once you are serving as an AC, please try to be more responsive than if you were a reviewer, as this could really help improve the entire review process. But also keep in mind that you will likely stumble upon unresponsive reviewers or emergency reviewers. Give yourself a lot of lead time and send 2x the email out at once if needed, because response takes time. Work with other ACs in your area might also help in the event of a emergency reviewer conflict of interest – you might end up being able to “exchange” emergency reviewers who are already responsive and willing to review.

• SACs. Please consider sharing as much resources as possible ahead of time with ACs, including training material (if any) and emergency reviewer info. This could help reduce your level of anxiety during the review process caused by unnecessary communication.

  My NAACL SACs did a great job at providing what is available, but I do wish there were a bit more training one could complete in prior so I can start the job more prepared.

• Program chairs. This is more of a comment about the reviewing software/features to consider, rather than about the chairs themselves, who I believe are doing a great job within their capacity. While the overall process was smooth for me, I find myself struggling from time to time with small issues that prohibited a potentially even smoother experience. I have personally experienced this in two places:

  The first involves transparency regarding withdrawn papers. The information about desk rejects and withdrawals during author response were either not shared until a few email exchanges later, or not available in a prominent interface to ACs/reviewers. A “simple” status change in the review system can easily result in hundreds of emails and the delay/confusion associated with them, given the size of our conferences nowadays.

  The second involves reviewer interaction and reviewer conflicts of interest. Never having been on the other end of the “chase” email, I had never anticipated having to send them one by one from my personal mailbox, filling paper information into the email template I drafted for myself. I was also surprised to find out that one of the reviewers never agreed to review in the first place. When it comes to emergency reviewers, there was no easy way to check for COI other than floating names to SACs, who are authorized to add them or tell us if there had been a COI. While the need for a centralized management of sensitive information is very much understandable, it is at these times I couldn’t help but think how much time a semi-automated system could save everyone.

Overall, I am excited to have had the opportunity to serve as an area chair, and to be able to share my experience that hopefully helps others. I am also encouraged that the field is actively taking measures to improve the review process (shout out to ACLRollingReview). Can’t wait to see how we can collectively improve the reviewing/publishing process in the next few months/years!