Reviewers: new and improved review process for ITiCSE 2017
Like SIGCSE 2017, ITiCSE 2017 is moving to an improved review process.
In the past, submissions were reviewed by a number of independent reviewers who had no way of communicating with one another. The program chairs had to decide whether to accept or reject each submission on the basis of a set of independent reviews that were sometimes widely divergent.
The new process for ITiCSE 2017 is an improvement in several respects.
- Reviewers will now "bid" for papers that they want to review, increasing the chance that they will review papers that interest them and where they can more effectively judge the contribution. If reviewers take the bidding process seriously, this should reduce the chance of weak reviews arising from a lack of familiarity with the content of a submission. As part of the bidding process, reviewers will be required to note papers with which they have a conflict of interest (e.g., from colleagues or project partners). Reviewers who bid for only a handful of papers, or who do not bid at all, should expect to be assigned the papers that no other reviewers have bid for.
- Once a review has been entered, the reviewer will be able to see the other reviews for the submission, but not the names of the other reviewers. They will then be able to discuss within EasyChair any differences in their perceptions of the submission. For submissions where the reviews present clearly divergent points of view, this discussion may bring the reviewers closer to consensus. They will then be able to modify their reviews, for example to take account of valid concerns - positive or negative - voiced by other reviewers.
- A number of Associate Program Chairs (APCs) will be appointed, and each submission will be assigned to one APC. The APCs will encourage discussion among the reviewers, especially for submissions with divergent reviews. They will then write a metareview, summarizing the reviews and providing a recommendation for the program chairs. This metareview will be sent to the authors along with the regular reviews.
- Papers that still have widely differing reviews will be discussed within EasyChair by the program chairs and the APCs.
- The final accept/reject decision will be based on the regular reviews and the metareview, and will also be influenced by the desire to have a broad yet coherent program. For example, single outlier submissions that do not fit readily into any session might be rejected even if their reviews are somewhat favorable.
We believe that the extra effort invested in this approach will lead to a better reviewing process, a better program, and greater acceptance of the reviewing process. We look forward to experiencing the new review process in action, evaluating it, and adjusting it if need be, and we hope that all of our reviewers and APCs will embrace it and help make it a success.
Paper bidding will take place shortly after the submission deadline in mid-January. There will then be three weeks for reviewing, followed by nearly two weeks for discussion and possible revision of reviews. The process should be over for reviewers by the third week in February, while APCs might be required to discuss papers with program chairs until the first week in March.
