← Blog

How to Write a Meta-Review?

The main thing to keep in mind when writing a meta-review is that it has two audiences: the authors, and a senior area chair (SAC). The authors want to know what to do with their paper next; the SAC wants to know what your recommendation is and whether they can trust it. The good news is that a single text can serve both, as long as you write with both in mind. Most meta-reviews have a 4-section structure: a summary, the strengths, the weaknesses, and the decision. Keep each of them direct—SACs read many of these. Also, keep each of them specific—SACs need to be able to trust you, and generic comments are hard to trust.

1. Summary
2. Strengths
3. Weaknesses
4. Your decision

Summary

Write a one-paragraph summary of the paper in your own words. Some forms require this; if yours doesn’t, do it anyway. Don’t assemble it out of the reviewers’ summaries—read the paper’s abstract, quickly skim through it (read it fully if necessary), and say what it does. For the authors, this shows the quality of their own writing: if skimming the paper is not enough to understand its high-level contribution, then the paper is not communicating it well. For the SAC, this paragraph should be self-contained. They should be able to read it and get a high-level idea of the paper—problem, method, main result—without opening the PDF.

Strengths

Be generous here about what is good about the paper. Summarise the strengths highlighted by the reviewers, say where they agreed and where they didn’t; also give your own opinion about each strength. For the authors, this is what tells them what to keep for the next version; even a paper you are rejecting should get a real strengths paragraph, and a meta-review without one reads as dismissive and is rarely accurate. For the SAC, this is your case for the paper: if you are recommending acceptance, this is where you explain why it is exciting and worth a slot. If the strongest thing you can write is that the paper is clearly written, that is informative too, and your recommendation should probably reflect it.

Weaknesses

Separate this section into three parts: true technical weaknesses, scope/excitement weaknesses, nice-to-haves. The true technical weaknesses should cover errors, unsupported claims, overstatements, and missing experiments which the paper’s main claim depends on. The scope/excitement weaknesses should cover things related to how interesting the research question is, how excited you and the reviewers feel about it, and how narrow it is. Finally, nice-to-haves cover non-critical improvements, e.g., more datasets, more models, more languages, extra ablations. Sometimes an ablation is essential, and sometimes it would only make the paper better and is not required for the paper to be complete: it is your job to distinguish between these.

If you explicitly split these weaknesses in your meta-review, the SAC can then immediately see what drove your decision, and the authors can see how to improve their paper. In both cases, remember there are people reading this: be constructive and respectful, there is no point in writing “this is a terrible paper”, and where you can, suggest how a problem might be fixed (if you can—it is not always clear how, and it’s fine to say so).

One thing both reviewers and ACs often skip, but which is important in the AI era: comment explicitly on the quality of the writing. Is the paper readable? If there is math in it, is the notation defined, are the theorems interpretable, and does the formalism actually serve a function? Don’t be afraid to write that you did not understand a section, or that a derivation is convoluted—you are a competent reader, and if you could not follow it, that is a fact about the paper and not about you. Besides that, a note on whether the paper looks AI-generated is also useful; I don’t think it’s a problem to get AI assistance when writing, but if the paper is unreadable, AI-like stylistic choices are a red flag.

Your decision

Be confident about rejecting papers. If accepting a paper requires you to believe a long list of changes promised in the rebuttal, then it is a rejection—you will not be able to check those changes, and neither will the reviewers. The authors can revise and resubmit in the next cycle, and if the changes are as good as promised, the paper will get in then. No lives are lost.

In ARR, Findings papers deserve some extra care, and it helps to distinguish two kinds of borderline paper.¹ The first is borderline in excitement, but sound and complete, with no clear path forward. Another round of work will not change these papers much, and Findings is a good home for them. The second is borderline but exciting, where what is missing is experiments the authors could actually run. These should be accepted, or rejected with a clear “improve and resubmit”; they should not go to Findings, because you would be freezing an exciting paper in its weakest version when one more cycle would have made it a strong one.

Finally, whatever you decide, state the arguments which actually drove your recommendation, especially when you go against the average of the scores—the SAC needs to be able to follow your reasoning.