Improved review process

Since last year, ITiCSE moved to an improved review process. In the past, submissions were reviewed by a number of independent reviewers who had no way of communicating with one another. The program chairs had to decide whether to accept or reject each submission on the basis of a set of independent reviews that were sometimes widely divergent.

The new process is an improvement in several respects.

1. Reviewers now "bid" for papers that they want to review, increasing the chance that they will review papers that interest them and where they can more effectively judge the contribution. If reviewers take the bidding process seriously, this should reduce the chance of weak reviews arising from a lack of familiarity with the content of a submission. As part of the bidding process, reviewers will be required to note papers with which they have a conflict of interest (e.g., from colleagues or project partners). Reviewers who bid for only a handful of papers, or who do not bid at all, should expect to be assigned the papers that no other reviewers have bid for.
2. Once a review has been entered, the reviewer will be able to see the other reviews for the submission, but not the names of the other reviewers. They will then be able to discuss within EasyChair any differences in their perceptions of the submission. For submissions where the reviews present clearly divergent points of view, this discussion may bring the reviewers closer to consensus. They will then be able to modify their reviews, for example to take account of valid concerns - positive or negative - voiced by other reviewers.
3. A number of Associate Program Chairs (APCs) will be appointed, and each submission will be assigned to one APC. The APCs will encourage discussion among the reviewers, especially for submissions with divergent reviews. They will then write a metareview, summarizing the reviews and providing a recommendation for the program chairs. This metareview will be sent to the authors along with the regular reviews.
4. Papers that still have widely differing reviews will be discussed within EasyChair by the program chairs and the APCs.
5. The final accept/reject decision will be based on the regular reviews and the metareview, and will also be influenced by the desire to have a broad yet coherent program. For example, single outlier submissions that do not fit readily into any session might be rejected even if their reviews are somewhat favorable.

We believe that the extra effort invested in this approach leads to a better reviewing process, a better program, and greater acceptance of the reviewing process.

Paper bidding will take place shortly after the submission deadline of paper abstracts on January 15, 2018. There will then be about three weeks for reviewing, followed by nearly two weeks for discussion and possible revision of reviews. The process should be over for reviewers by the last week in February, while APCs might be required to discuss papers with program chairs until the first week in March.

Important Note

Following the SIGCSE conference policies, reviewing for papers is double-blind, including protection of the complete anonymity of the APCs and reviewers during the discussion phase. Also, only polished work is accepted as there is no review-revise-review cycle.

Summary of commitment: Reviewers and APCs

Tuesday 16 January - Monday 22 January (1 week)

All reviewers and APCs bid for papers.

If a reviewer or APC fails to bid within the allocated week, it will be assumed that they have dropped out of contact.

Thursday 25 January - Wednesday 14 February (3 weeks)

Reviewers review papers.

APCs monitor reviews, respond to issues, form opinions of the papers assigned to them.

Thursday 15 February - Saturday 24 February (10 days)

Reviewers and APCs discuss papers on EasyChair, possibly moving closer to consensus.

APCs write metareviews and recommend outcomes.

Optional: volunteer reviewers review additional papers at short notice.

Sunday 25 February - Tuesday 6 March (10 days)

APCs might be required to discuss particular papers with program chairs.

Saturday 17 March - Friday 23 March (1 week)

Optional: volunteer reviewers review submissions for posters and for tips, techniques, and courseware.

Additional Information: Reviewers and APCs

Based on ITiCSE 2017, we anticipate that reviewers will be asked to review about 5 papers and APCs to oversee and metareview about 12 papers. The final numbers will depend on the number of reviewers, the number of APCs, and the number of submissions. Requests for lighter loads cannot be accommodated; if you cannot take the full load, please do not volunteer your services.

All reviewers and APCs are expected to engage with the process throughout the six weeks from Monday 15 January to Sunday 25 February, as described below. Those who cannot commit to engagement throughout this period should not volunteer to be reviewers or APCs for ITiCSE 2018.

All of the dates listed below are taken to be in the AoE (Anywhere on Earth) timezone, UTC-12, and all deadlines are at 23.59 UTC-12 on the specified date. If you are not sure what UTC-12 means, you will not be late if you regard the deadlines as in your own timezone.

With a conference the size of ITiCSE, it is impossible to keep in close contact with all reviewers and all APCs at all times. People drop out of contact for many reasons, including illness, untoward weather events, technological problems, and failure to ensure that their email system does not treat EasyChair messages as spam or junk. The system must have the flexibility to deal with these losses, and the descriptions below will explain how this is managed.

Tuesday 16 January - Monday 22 January (1 week): bidding

All reviewers and APCs bid for papers.

Papers are assigned to reviewers and APCs not on the basis of topic overlaps but on the basis of their bids, or expressions of interest. The bidding process is how you indicate which papers you would like to review and which papers you are willing to review.

It is important to understand that bidding does not mean choosing the papers you will review. That choice will be made by EasyChair, using an algorithm that tries to allocate the papers reasonably fairly. Ideally, bidding is a process of identifying all of the papers that you would like to review or are willing to review. The more papers you say you can bid, the more likely you are to be allocated papers from your list.

Bidding on just a small number of papers influences the algorithm unfairly in your favour. To counter this, anyone who bids on too few papers will have their bids supplemented with an arbitrary selection of papers. If you do not wish this to happen to you, please ensure that you bid 'yes' or 'maybe' for at least 30 papers, with at least 15 of those being 'yes'.

If a reviewer or APC fails to bid within the allocated week, it will be assumed that they have dropped out of contact; they will be removed from the review team and not allocated any papers for ITiCSE 2018.

Once you are notified that bidding is open, you should log in to your EasyChair account and access the ITiCSE 2018 conference with the appropriate role. Reviewers are called ' program committee' on EasyChair, and APCs are called 'senior reviewers'.

You are likely to find yourself on a page listing the Submissions to the conference, but this is not the page you want. Instead click on the Paper Bidding menu. Again you will see a list of the submissions, but for each one you can now click links to indicate the level of your interest in reviewing them. If you wish to see the abstracts as well as the titles, click either Show full abstracts or Show abstract summaries at the upper right of the screen.

There is no need to click No for papers you do not wish to review; No is the default option. You should click Conflict for a paper if you know who wrote it, and thus cannot be sure of reviewing it impartially.

The papers are presented in a random order for each bidder, to help ensure a good spread of bids. However, this order is preserved for each bidder, so if you spread your bidding over several sessions, the papers will appear in the same order each time you access them.

The bids are colour-coded, but the selected yes/maybe/no/conflict link is also indicated in bold face, which will help those who have trouble distinguishing the colours.

Thursday 25 January - Wednesday 14 February (3 weeks): reviewing

A few days after the close of bidding, reviewers and APCs will be notified which papers they have been assigned, and paper reviewing will begin.

When you log in to the appropriate role on EasyChair you will now see a Reviews menu. On this page you can download the papers assigned to you. You should also click Add Review for one of the papers, to get a first look at the review form; just do not submit the review!

Here are some points that you should consider when reviewing.

1. Your review should be written by you; please do not farm your reviews out to other people. If you believe that other people should be reviewing for ITiCSE, please encourage them to do so in their own right. If you do not have the time or inclination to do your own reviewing for ITiCSE, please do not volunteer to do so.
2. Your review should indicate whether the paper as submitted is acceptable for ITiCSE. There is no review-revise-review cycle, so papers should be marked as accept if they are currently acceptable, not if they have the potential to become acceptable after substantial revision. In particular, there is no such thing as a conditional accept. If a paper is not suitable for acceptance subject to minor revisions, it should be rejected.
3. Whether or not a paper is accepted, the authors will want to know why; please ensure that your review gives reasons, explanations, examples of the points you make. Reviews just a few lines long are generally not helpful to the authors, the APCs, or the program chairs. In particular, on reading the text parts of a review, authors should understand why the paper was given the overall score that it was.
4. Be sure that your overall recommendation tallies with your impression of the paper. If you do not believe that the paper should be accepted, please do not give it a 4 just to be nice, or to err on the side of generosity. It is more helpful to the program chairs, the APCs, and the authors if your review is honest and transparent and your overall recommendation reflects your impression.
5. ITiCSE reviewing is double-blind. You should not know who wrote the papers that you review, and the authors should not know who reviewed it. Please therefore take care not to divulge your identity in your review, for example by saying ' This paper that I wrote might be helpful to you.'
6. Even if EasyChair permits you to add an attachment to your review, please do not do so. Your review should be contained entirely within the review form.
7. Most papers have some flaws in their writing. The reviewers are not required to be editors, pointing out every flaw that requires correction. However, some reviewers choose to do this, and many authors appear to appreciate the additional service.
8. It is a good idea for reviews to be grammatically correct - especially if they criticise the grammar of the paper. Just as a poorly written paper gives the reader a poor first impression, a poorly written review can make the authors less inclined to accept its content. Please read your own reviews carefully, and, if necessary, submit revised versions with the spelling and grammar errors corrected. If you choose to draw attention to spelling issues in a paper, please be aware that many words have different spellings in different regions. Neither 'analyze' nor 'analyse' is wrong: they are different acceptable spellings; likewise 'program' and 'programme', 'behavior' and 'behaviour', and many more such pairs.
9. ITiCSE is not a research conference, and reviews should not say that it is. It certainly accepts research papers, but it also accepts practice papers. Submissions should not be rejected solely because they do not present empirical research. The specification says that types of paper might include "experience reports describing a classroom innovation and an assessment of its impact"; but nowhere does it explicitly exclude any type of paper. An experience report might be acceptable if it provides some evidence of the success or failure of the experience; but it might equally be acceptable if it provides a sound explanation as to why the experience is expected to be successful. If you recommend that a paper be rejected, please be clear that it is because of the quality of the paper or its contribution, not because it is not a research paper.
10. Good papers will be relevant to an international audience. ITiCSE certainly accepts papers based on work from specific countries, but such papers are enhanced if they have clear lessons for people from other countries. For example, a paper that writes about making it easier for people from the mountain regions to succeed in a computing degree would be better received if it made a clear generalisation to under-represented minorities - or perhaps to people with better lung capacity - regardless of country.
11. The limit of 6 pages is a maximum, not a minimum. There is nothing wrong with shorter papers, if they are good and pertinent; they do not need to be padded out to six pages. At the same time, if a shorter paper clearly lacks something that would have strengthened it, feel free to point that out to the authors.
12. It is acceptable to point out formatting issues, but not to make them the deciding factor in a recommendation to reject. Also, please be aware that some authors are still struggling to come to grips with the new ACM templates. If in doubt, leave it to the proceedings chair to decide on a paper's compatibility with the template.

Please try to complete your reviews before the deadline; it is a deadline, not a target submission time. When reviewers leave the task till the last minute, things can go wrong (illness, work emergency, family emergency, reviewing taking longer than you anticipated), and then the reviews are not done by the deadline - which means that we are left asking for volunteers to review extra papers at very short notice.

There will be periodic reminders during the review period. If at any time during this period you become aware that you will not have your reviews written by the deadline, please contact us immediately so that we can decide how to deal with it.

Once you have submitted your review for a paper, you will be able to see the other reviews already submitted for that paper, and EasyChair will notify you by email when further reviews are submitted. Please consider these other reviews, in preparation for the discussion period.

APCs, you are also expected to watch the reviews for your assigned papers as they come in, but you will need to explicitly add them to your watchlist so that you will receive email notifications. From the ITiCSE 2018 menu choose My watchlist, choose the Select all option, then Save watchlist. Please keep an eye open for obvious problems: a review might have been submitted for the wrong paper, or might identify the author or the reviewer, or might be so brief as to be unhelpful. We will do our best to notice such problems, but your attention will also help.

Thursday 15 February - Saturday 24 February (10 days): discussion, revision, and metareviewing; optional emergency reviewing

It is inevitable that some reviewers will not complete their reviews by the deadline. Once the deadline has passed, reviewers who have not submitted their reviews, and have not contacted the database coordinators to discuss this, will be dropped from the review team in the assumption that they have fallen out of contact for some reason.

We will then be seeking emergency reviewers who are willing and able to complete one or two more reviews at very short notice. This is very much a voluntary task; no reviewer is expected to undertake emergency reviews; but those who do so are greatly appreciated.

The discussion is a required aspect of the reviewing task. As a reviewer, you are expected to carefully consider, in the light of the other reviews for a paper, whether your own review is a reasonable one, and to join the other reviewers and the APC in discussing the paper.

In most cases the discussion will be led by the APC. If there is clear consensus on a paper, the APC might let the reviewers know that it requires no discussion. Of course, the reviewers may discuss a paper even if there is a consensus, but in that case discussion is not required.

Consensus among reviewers is not essential, but it helps to persuade the authors that the reviewers know what they're writing about. Consensus does not mean identical overall scores or absolute agreement on all points, but it probably means a spread of no more than 2 in the overall evaluations. If a paper's evaluations include a 1 and a 6, or even a 2 and a 5, there is clear scope for the outlying reviewers to try to persuade one another of the validity of their points.

Consensus should not necessarily move toward the majority viewpoint if there is one. If one reviewer notices an aspect of a paper that suggests a different outcome from the majority of reviewers, that reviewer should try to persuade the others to reconsider - and the others should consider the arguments carefully.

Disparity in reviews can sometimes be brought about by different focuses. For example, one review might recommend acceptance because of the novelty of the topic while another might recommend rejection because of flaws in the data collection or analysis. In such cases, the discussion might focus on which criterion matters more for ITiCSE.

In the end, if there are differences that cannot be resolved, the reviews should reflect that. We are not asking reviewers to present a unified front at all costs; we are asking you to more or less agree about a paper if that is possible.

Discussion takes place by way of comments. When you are viewing the reviews and comments for a paper, you add a comment by using the Add comment field at the bottom of the screen. The reviewers and APCs are all mutually anonymous, so please do not divulge your identity in a comment. However, to let the other participants know which reviewer is saying what, please identify yourself and one another by review number. For example, you might write "R4 here: APC, I see your point, but I still think R3 makes a very good case that the required changes would make the paper too different from the one that was submitted for review." You will appreciate that there is little point in writing simply "I agree" when different people have expressed opposing thoughts. Which one are you, and which one are you agreeing with?

Comments on a paper are seen only by the other reviewers for that paper, the APC for that paper, and the program chairs. In particular, they are not seen by the paper's authors. For this reason, if a point arises during the discussion that would benefit the authors, you should revise your review to include it. Likewise, if in the discussion you change your mind about points you have made, or about your overall impression of the paper, you must submit a revised review that expresses your new viewpoint. However, please ensure that your review stands in its own right; it is not appropriate to make a point by referring the authors to another review.

When revising a review, please ensure that the written comments and the numerical scores still match.

APCs, by about halfway through the discussion period you should have a good idea about your overall recommendation for each paper. Your metareview will consist of an accept/reject recommendation, a text field that will be seen by the paper's authors, and a confidential text field that in most cases will not be needed. If the reviews for a paper are in reasonable agreement, your metareview might simply note this. If there is some disagreement, your metareview might refresh the important points and explain the reasons for your recommendation. If you post your metareview before the end of the discussion period, the reviewers can comment on it, too, if they wish, and you will have the opportunity to revise it.

Once the discussion and metareview period is over, the program chairs will make the decision for each paper. Please remember that they must consider many factors when making these decisions, and that they might not always make the decision recommended by the APC. Also, the program chairs might wish to discuss particular papers with individual APCs.

Please be aware that we will be looking informally at the performance of all reviewers. Factors considered will be the quality of their initial reviews, their participation in the discussion, and their willingness to change their position on a paper if it seems appropriate.

Optional: Saturday 17 March - Friday 23 March: reviewing submissions for posters and for tips, techniques, and courseware

In the week beginning 17 March we will require reviewers for posters and for tips, techniques, and courseware. There will be far fewer of these submissions, they will be shorter than the full paper submissions, and they will generally be much easier to review. When we know how many of these submissions there are, we will call for volunteers to undertake this additional reviewing.

These reviews will have no APCs and no discussion period; they will be considered directly by the chairs for their respective tracks.