Reviewer Instructions for Full Papers

Introduction

Thank you for agreeing to review papers for NLDL, which aims to showcase high-quality research in the field of machine learning. We value your expertise and commitment to ensuring the quality of the papers submitted. In this guideline, we outline key considerations and criteria for evaluating submissions.

Review

The review form requires you to state:

• Summary of the paper: You should provide a concise description of the paper.  In a couple sentences, you have to state what is the paper about, what it is proposing, and the main results of it.  Provide a factual representation of the paper and the author claims.
  The objective of this summary is two-fold.  One one hand, the authors can see if the submission was understood as intended, and address any factual errors.  On the other, the Area Chair can use this information to obtain a general picture of the submission.

• Strengths: You should provide a thorough assessment of the positive points of the paper.  In particular, how it fits within the expected criteria (see below).  You should think whether the paper makes a contribution to the knowledge of the field, and whether this increment .  In other words, whether the paper makes a positive step towards new knowledge or understanding of the existing one.

• Weaknesses:  You should provide a thorough assessment of the negative points of the paper and its limitations.  In particular, you should focus on describing problems with the paper, e.g., related to the theory, theoretical framework, experimental setup, or experimental results.  Missing literature and links to existing methods should be described as well, although they shouldn't be the main point to reject the paper.

• Rating (hidden to the ACs and other reviewers): You should provide a score based on the following scale below.  You cannot give borderline ratings (3).  Clear signals towards accepting (4 or more) or rejecting the paper (2 or less) are better for the discussion and to guide a final decision among the reviewers.  In rare cases where you have these borderline evaluations, you should have a thorough description articulating why you consider the paper as borderline.  Regardless, you should give a recommendation towards accept or reject. 

  • 5: Clear accept, definitely interesting results for the ML community

  • 4: Accept, contain novelty worth sharing

  • 2: Reject, contributions are unclear/non-convincing or needs more experiments to back up the results

  • 1: Clear reject, the submission is too weak or out of scope

Note that in this instance, the Area Chairs and other reviewers do not have access to the ratings and will make a decision based on the descriptions alone.

• Justification: You should provide a thorough description of why you rate the paper the way you did.  You should not say "see the strengths or weaknesses."  This statement is unhelpful.  Instead, you should summarize and raise the main points that made you lean towards your rating.  You should justify with summaries the main relevant points, and how they outweigh the others.  For example, if you recommend a rejection, you should raised and summarize the main contributions and strengths of the paper and describe how they are not sufficient to outweigh the limitations.  While you do so, you should also summarize what the main problems are.
  The goal of this description is for the Area Chair to understand your rationale behind the evaluation of the paper.  Note that in this instance, the Area Chairs do not have access to the ratings and will make a decision based on the descriptions alone.

Post-rebuttal Review

After the reviews has been published, during the rebuttal phase, the authors will have the opportunity to write a rebuttal in the forums to respond to your questions and resubmit a revised version of the paper addressing your questions and problems.  There will be no discussion phase during the rebuttal. However, we encourage the reviewers to ask for further clarifications and reply to the authors rebuttal as needed.  The comments will be open in case the authors have more questions about the review, but the reviewers are not forced to interact.  

Final Decisions

After the "Post-rebuttal Review" phase, you should read all the reviews, rebuttal discussions, and other comments.  You must discuss with the other reviewers and the Area Chair to reach an agreement (if possible).  You will be able to update the rating, and should write a post-rebuttal justification.

General Review Criteria

1. Coherence and Correctness: The primary criterion for acceptance is the clarity and correctness of the paper. Ensure that the paper is logically structured and that the presented methodology, experiments, and results are sound. Identify any errors or inconsistencies, and provide constructive feedback for improvement.

2. Incremental Contributions: Recognize that not all papers need to introduce groundbreaking innovations. Incremental advances in existing knowledge are valuable contributions to the field. Assess whether the paper adds meaningful insights, even if they build upon prior work. In this sense, incremental work is not a basis for rejection.

3. Application Papers: We welcome papers that focus on real-world applications of machine learning. However, such papers must meet the following additional criteria:
   a. Non-Trivial Challenges: The paper should clearly articulate the challenges faced in the application domain and explain why solving these challenges is not trivial. Evaluate the extent to which the authors demonstrate a deep understanding of the specific domain problems.
   b. Solution Description: Evaluate the proposed solution's novelty and effectiveness in addressing the identified challenges. Assess whether the solution has practical relevance and can be applied beyond the specific case studied.

4. Reproducibility: Assess the paper's reproducibility by evaluating whether the authors provide sufficient information, code, and data to replicate their experiments. Highlight any deficiencies in reproducibility and suggest improvements.

5. Clarity and Presentation: Evaluate the clarity and organization of the paper. Ensure that the writing is clear, concise, and well-structured. Identify areas where the authors can improve explanations, figures, and tables to enhance the paper's comprehensibility.

6. Ethical Considerations: Check for ethical considerations related to data collection, use, and potential societal impacts. If ethical concerns are identified, make sure they are addressed or discussed appropriately in the paper.

Specific Review Guidelines

7. Novelty: Assess the extent to which the paper advances the state of the art or provides valuable insights. Even incremental papers have some novelty (the delta is the increment over the original paper), you should make sure that this delta is enough to constitute a gain in knowledge. 

8. Experimental Rigor: Evaluate the experimental design, methodology, and statistical analysis. Ensure that experiments are well-designed, reproducible, and draw meaningful conclusions.

9. Related Work: Check if the paper adequately discusses and contextualizes related work. Ensure that the authors acknowledge prior research and explain how their work differs or builds upon it.

10. Discussion of Limitations: Assess whether the paper openly discusses its limitations. Encourage authors to address shortcomings and provide suggestions for future work to mitigate these limitations.

11. Significance: Consider the significance of the paper's contributions to the machine learning community and potential practical applications. Assess whether the work has broader implications beyond the specific problem domain.

12. Citation and Attribution: Ensure that authors properly attribute prior work, data sources, and collaborators. Check for any potential conflicts of interest or undisclosed affiliations.

Reviewer's Summary and Recommendations

In your review, please provide a concise summary of your evaluation and specific recommendations for the paper. Summarize the strengths and weaknesses, and justify your final recommendation. Constructive feedback is essential to help authors improve their work.

Remember that our primary goal is to maintain a high standard of quality and rigor in machine learning research while being open to both novel contributions and practical applications. Your expertise and thorough evaluation are invaluable in achieving this goal.

Thank you for your dedication to advancing the field of machine learning.

Resources for reviewing

• Daniel Dennet, Criticising with Kindness.

• Niklas Elmqvist, Mistakes Reviewers Make

• Views from multiple reviewers: Last minute reviewing advice

• CVPR 2024 Reviewer Tutorial Slides

FAQ

About the review process

• Q. I found that the submission is not anonymized correctly.  What should I do?
  A. Contact the Area Chair and the Program Chairs immediately.  You can send a message through OpenReview and put both groups as readers.  If you have not received a response through OpenReview in two days, please contact through email.

• Q. Will there be a discussion phase between the authors and reviewers for the rebuttal?
  A. No.  The authors will submit their rebuttals and comments during the rebuttal phase.  The reviewers are not expected to interact with the authors, but you are encourage to do it. 
  After the rebuttal, the reviewers must discuss the rebuttal and reviews with the other reviewers and the Area Chair.  Afterwards, the reviewers are expected to update their scores and submit a new justification for the final score.

• Q. I won't change my score, do I have to write a post-rebuttal justification?
  A. YES.  You should write why the authors rebuttal and revised manuscript didn't address your concerns.  Note that just saying "the authors' comments didn't addressed my concerns" is not a valid justification.  You should elaborate on why the answers did not address your concerns and what you were expecting instead.

About LLMs

• Q. What is the LLM Policy for reviewers?
  A. Reviewers may use any device, including an LLM, to polish their review wording, but must vouch for, and be responsible for, the accuracy of the review. It is a significant act of reviewer misconduct to allow an LLM to see a submission. PCs interpret showing a submission to an LLM as a deliberate reviewer violation of confidentiality. The PCs reserve the right to report reviewer misconduct to future machine learning and related conferences. These conferences then may take actions, e.g., there was a recent PAMI-TC vote that CVPR reviewer misconduct may lead to a 2-year submission ban.

• Q. How will the LLM policy be implemented?
  A. An author may complain to their AC that a summary (and/or other parts of the review) have been prepared by an LLM that has seen the paper. Such a complaint would need to be supported by an example summary (or other part of the review) prepared by the author giving the paper to an LLM. If this matches the reviewer’s comments sufficiently, ACs will pass the complaint on to PCs who are then entitled, but not required, to act. Complaints must not appear on the rebuttal, but be submitted on a separate form. PCs strongly discourage frivolous complaints. Authors should be aware that a complaint to an AC about a review prepared by an LLM without reasonable evidence in support of that complaint, is wasting the ACs time.