Reviewer Guidelines for NLDL

Introduction

Thank you for agreeing to review papers for NLDL, which aims to showcase high-quality research in the field of machine learning. We value your expertise and commitment to ensuring the quality of the papers submitted. In this guideline, we outline key considerations and criteria for evaluating submissions.

General Review Criteria

1. Coherence and Correctness: The primary criterion for acceptance is the clarity and correctness of the paper. Ensure that the paper is logically structured and that the presented methodology, experiments, and results are sound. Identify any errors or inconsistencies, and provide constructive feedback for improvement.

2. Incremental Contributions: Recognize that not all papers need to introduce groundbreaking innovations. Incremental advances in existing knowledge are valuable contributions to the field. Assess whether the paper adds meaningful insights, even if they build upon prior work. In this sense, incremental work is not a basis for rejection.

3. Application Papers: We welcome papers that focus on real-world applications of machine learning. However, such papers must meet the following additional criteria:
   a. Non-Trivial Challenges: The paper should clearly articulate the challenges faced in the application domain and explain why solving these challenges is not trivial. Evaluate the extent to which the authors demonstrate a deep understanding of the specific domain problems.
   b. Solution Description: Evaluate the proposed solution's novelty and effectiveness in addressing the identified challenges. Assess whether the solution has practical relevance and can be applied beyond the specific case studied.

4. Reproducibility: Assess the paper's reproducibility by evaluating whether the authors provide sufficient information, code, and data to replicate their experiments. Highlight any deficiencies in reproducibility and suggest improvements.

5. Clarity and Presentation: Evaluate the clarity and organization of the paper. Ensure that the writing is clear, concise, and well-structured. Identify areas where the authors can improve explanations, figures, and tables to enhance the paper's comprehensibility.

6. Ethical Considerations: Check for ethical considerations related to data collection, use, and potential societal impacts. If ethical concerns are identified, make sure they are addressed or discussed appropriately in the paper.

Specific Review Guidelines

7. Novelty: While we accept incremental papers, they should still contribute something to the field (either an empirical or theoretical insights). Assess the extent to which the paper advances the state of the art or provides valuable insights.

8. Experimental Rigor: Evaluate the experimental design, methodology, and statistical analysis. Ensure that experiments are well-designed, reproducible, and draw meaningful conclusions.

9. Related Work: Check if the paper adequately discusses and contextualizes related work. Ensure that the authors acknowledge prior research and explain how their work differs or builds upon it.

10. Discussion of Limitations: Assess whether the paper openly discusses its limitations. Encourage authors to address shortcomings and provide suggestions for future work to mitigate these limitations.

11. Significance: Consider the significance of the paper's contributions to the machine learning community and potential practical applications. Assess whether the work has broader implications beyond the specific problem domain.

12. Citation and Attribution: Ensure that authors properly attribute prior work, data sources, and collaborators. Check for any potential conflicts of interest or undisclosed affiliations.

Reviewer's Summary and Recommendations

In your review, please provide a concise summary of your evaluation and specific recommendations for the paper. Summarize the strengths and weaknesses, and justify your final recommendation. Constructive feedback is essential to help authors improve their work.

Remember that our primary goal is to maintain a high standard of quality and rigor in machine learning research while being open to both novel contributions and practical applications. Your expertise and thorough evaluation are invaluable in achieving this goal.

Thank you for your dedication to advancing the field of machine learning.

Area Chair Guidelines for NLDL

Introduction

As an Area Chair for NLDL, your role is pivotal in ensuring the quality and fairness of the paper selection process. This guideline is designed to provide you with clear instructions and criteria to guide your decision-making and interactions with reviewers. Please ensure that you align your decisions with the conference's objectives and the expectations outlined in the reviewer guidelines.

General Responsibilities

1. Paper Assignment: Assign papers to reviewers with expertise in the respective areas, ensuring that each submission receives a fair and expert evaluation.

2. Expertise Match: Verify the expertise of reviewers and make adjustments if necessary to ensure that they are qualified to evaluate the assigned papers.

3. Communication: Maintain clear and timely communication with reviewers and authors, facilitating discussions as needed while preserving reviewer anonymity.

4. Quality Control: Oversee the review process to ensure that reviewers adhere to the conference's guidelines and standards.

Specific Area Chair Guidelines

Coherence and Correctness

1. Primary Criteria: Emphasize that the primary criterion for acceptance is the coherence and correctness of the papers. Encourage reviewers to thoroughly assess the clarity and logical soundness of the submissions.

2. Quality Assurance: Review reviewer comments to identify potential discrepancies or concerns related to correctness and coherence. Address any discrepancies through reviewer discussions if necessary.

Incremental Contributions

3. Clarification: Clarify that incremental contributions are acceptable and valuable but must still represent a meaningful advance in knowledge. Encourage reviewers to evaluate the significance of incremental contributions within the context of the field.

4. Contextualization: Encourage reviewers to assess how well the paper positions its work within the existing literature. Ensure that reviewers appreciate the significance of incremental advancements in knowledge.

Application Papers

5. Challenge Identification: Stress the importance of evaluating application papers based on the non-trivial challenges they address in the specific domain. Ask reviewers to carefully assess whether the challenges are clearly articulated.

6. Solution Evaluation: Ensure that reviewers critically evaluate the novelty, effectiveness, and practical relevance of the proposed solutions to address the identified challenges.

7. Discussion: Encourage reviewers to assess whether the paper effectively explains why the solutions to the application challenges are not trivial and how they contribute to the broader understanding of machine learning.

Reconciliation of Reviewer Feedback

8. Discrepancies: If reviewers provide conflicting recommendations, facilitate a discussion to reach a consensus. Engage reviewers in a constructive dialogue to address differing viewpoints and consider additional expert opinions if necessary.

9. Reviewer Feedback: Consolidate reviewer feedback, emphasizing key points and suggestions for authors to improve their submissions.

Ethical Considerations

10. Ethical Review: Remind reviewers to consider ethical aspects, such as data privacy, fairness, and potential societal impacts, and assess whether these aspects are appropriately addressed or discussed in the papers.

Recommendations to Authors

11. Clear Guidance: Provide clear, constructive guidance to authors based on the reviewer feedback. Encourage authors to address reviewer concerns, improve clarity, and enhance the quality of their submissions.

12. Consistency: Ensure that the feedback and recommendations to authors align with the conference's guidelines and criteria, emphasizing the importance of coherence, correctness, significance, and ethical considerations.

Final Decision

13. Decision Justification: Make well-informed decisions based on reviewer feedback and discussions. Justify your decisions in terms of the paper's quality, adherence to guidelines, and potential contribution to the field.

Conclusion

Your role as an Area Chair is integral to the success of our conference. By adhering to these guidelines and working closely with reviewers and authors, you will help maintain the conference's high standards of quality and fairness while promoting both incremental contributions and impactful application papers in the field of machine learning.

Thank you for your commitment to advancing the research in this domain.