Since last year, ITiCSE moved to an improved review process. In the past, submissions were reviewed by a number of independent reviewers who had no way of communicating with one another. The program chairs had to decide whether to accept or reject each submission on the basis of a set of independent reviews that were sometimes widely divergent.
We believe that the extra effort invested in this approach leads to a better reviewing process, a better program, and greater acceptance of the reviewing process.
Paper bidding will take place shortly after the submission deadline of paper abstracts on January 15, 2018. There will then be about three weeks for reviewing, followed by nearly two weeks for discussion and possible revision of reviews. The process should be over for reviewers by the last week in February, while APCs might be required to discuss papers with program chairs until the first week in March.
Following the SIGCSE conference policies, reviewing for papers is double-blind, including protection of the complete anonymity of the APCs and reviewers during the discussion phase. Also, only polished work is accepted as there is no review-revise-review cycle.
Tuesday 16 January - Monday 22 January (1 week)
All reviewers and APCs bid for papers.
If a reviewer or APC fails to bid within the allocated week, it will be assumed that they have dropped out of contact.
Thursday 25 January - Wednesday 14 February (3 weeks)
Reviewers review papers.
APCs monitor reviews, respond to issues, form opinions of the papers assigned to them.
Thursday 15 February - Saturday 24 February (10 days)
Reviewers and APCs discuss papers on EasyChair, possibly moving closer to consensus.
APCs write metareviews and recommend outcomes.
Optional: volunteer reviewers review additional papers at short notice.
Sunday 25 February - Tuesday 6 March (10 days)
APCs might be required to discuss particular papers with program chairs.
Saturday 17 March - Friday 23 March (1 week)
Optional: volunteer reviewers review submissions for posters and for tips, techniques, and courseware.
Based on ITiCSE 2017, we anticipate that reviewers will be asked to review about 5 papers and APCs to oversee and metareview about 12 papers. The final numbers will depend on the number of reviewers, the number of APCs, and the number of submissions. Requests for lighter loads cannot be accommodated; if you cannot take the full load, please do not volunteer your services.
All reviewers and APCs are expected to engage with the process throughout the six weeks from Monday 15 January to Sunday 25 February, as described below. Those who cannot commit to engagement throughout this period should not volunteer to be reviewers or APCs for ITiCSE 2018.
All of the dates listed below are taken to be in the AoE (Anywhere on Earth) timezone, UTC-12, and all deadlines are at 23.59 UTC-12 on the specified date. If you are not sure what UTC-12 means, you will not be late if you regard the deadlines as in your own timezone.
With a conference the size of ITiCSE, it is impossible to keep in close contact with all reviewers and all APCs at all times. People drop out of contact for many reasons, including illness, untoward weather events, technological problems, and failure to ensure that their email system does not treat EasyChair messages as spam or junk. The system must have the flexibility to deal with these losses, and the descriptions below will explain how this is managed.
Tuesday 16 January - Monday 22 January (1 week): bidding
All reviewers and APCs bid for papers.
Papers are assigned to reviewers and APCs not on the basis of topic overlaps but on the basis of their bids, or expressions of interest. The bidding process is how you indicate which papers you would like to review and which papers you are willing to review.
It is important to understand that bidding does not mean choosing the papers you will review. That choice will be made by EasyChair, using an algorithm that tries to allocate the papers reasonably fairly. Ideally, bidding is a process of identifying all of the papers that you would like to review or are willing to review. The more papers you say you can bid, the more likely you are to be allocated papers from your list.
Bidding on just a small number of papers influences the algorithm unfairly in your favour. To counter this, anyone who bids on too few papers will have their bids supplemented with an arbitrary selection of papers. If you do not wish this to happen to you, please ensure that you bid 'yes' or 'maybe' for at least 30 papers, with at least 15 of those being 'yes'.
If a reviewer or APC fails to bid within the allocated week, it will be assumed that they have dropped out of contact; they will be removed from the review team and not allocated any papers for ITiCSE 2018.
Once you are notified that bidding is open, you should log in to your EasyChair account and access the ITiCSE 2018 conference with the appropriate role. Reviewers are called ' program committee' on EasyChair, and APCs are called 'senior reviewers'.
You are likely to find yourself on a page listing the Submissions to the conference, but this is not the page you want. Instead click on the Paper Bidding menu. Again you will see a list of the submissions, but for each one you can now click links to indicate the level of your interest in reviewing them. If you wish to see the abstracts as well as the titles, click either Show full abstracts or Show abstract summaries at the upper right of the screen.
There is no need to click No for papers you do not wish to review; No is the default option. You should click Conflict for a paper if you know who wrote it, and thus cannot be sure of reviewing it impartially.
The papers are presented in a random order for each bidder, to help ensure a good spread of bids. However, this order is preserved for each bidder, so if you spread your bidding over several sessions, the papers will appear in the same order each time you access them.
The bids are colour-coded, but the selected yes/maybe/no/conflict link is also indicated in bold face, which will help those who have trouble distinguishing the colours.
Thursday 25 January - Wednesday 14 February (3 weeks): reviewing
A few days after the close of bidding, reviewers and APCs will be notified which papers they have been assigned, and paper reviewing will begin.
When you log in to the appropriate role on EasyChair you will now see a Reviews menu. On this page you can download the papers assigned to you. You should also click Add Review for one of the papers, to get a first look at the review form; just do not submit the review!
Here are some points that you should consider when reviewing.
Please try to complete your reviews before the deadline; it is a deadline, not a target submission time. When reviewers leave the task till the last minute, things can go wrong (illness, work emergency, family emergency, reviewing taking longer than you anticipated), and then the reviews are not done by the deadline - which means that we are left asking for volunteers to review extra papers at very short notice.
There will be periodic reminders during the review period. If at any time during this period you become aware that you will not have your reviews written by the deadline, please contact us immediately so that we can decide how to deal with it.
Once you have submitted your review for a paper, you will be able to see the other reviews already submitted for that paper, and EasyChair will notify you by email when further reviews are submitted. Please consider these other reviews, in preparation for the discussion period.
APCs, you are also expected to watch the reviews for your assigned papers as they come in, but you will need to explicitly add them to your watchlist so that you will receive email notifications. From the ITiCSE 2018 menu choose My watchlist, choose the Select all option, then Save watchlist. Please keep an eye open for obvious problems: a review might have been submitted for the wrong paper, or might identify the author or the reviewer, or might be so brief as to be unhelpful. We will do our best to notice such problems, but your attention will also help.
Thursday 15 February - Saturday 24 February (10 days): discussion, revision, and metareviewing; optional emergency reviewing
It is inevitable that some reviewers will not complete their reviews by the deadline. Once the deadline has passed, reviewers who have not submitted their reviews, and have not contacted the database coordinators to discuss this, will be dropped from the review team in the assumption that they have fallen out of contact for some reason.
We will then be seeking emergency reviewers who are willing and able to complete one or two more reviews at very short notice. This is very much a voluntary task; no reviewer is expected to undertake emergency reviews; but those who do so are greatly appreciated.
The discussion is a required aspect of the reviewing task. As a reviewer, you are expected to carefully consider, in the light of the other reviews for a paper, whether your own review is a reasonable one, and to join the other reviewers and the APC in discussing the paper.
In most cases the discussion will be led by the APC. If there is clear consensus on a paper, the APC might let the reviewers know that it requires no discussion. Of course, the reviewers may discuss a paper even if there is a consensus, but in that case discussion is not required.
Consensus among reviewers is not essential, but it helps to persuade the authors that the reviewers know what they're writing about. Consensus does not mean identical overall scores or absolute agreement on all points, but it probably means a spread of no more than 2 in the overall evaluations. If a paper's evaluations include a 1 and a 6, or even a 2 and a 5, there is clear scope for the outlying reviewers to try to persuade one another of the validity of their points.
Consensus should not necessarily move toward the majority viewpoint if there is one. If one reviewer notices an aspect of a paper that suggests a different outcome from the majority of reviewers, that reviewer should try to persuade the others to reconsider - and the others should consider the arguments carefully.
Disparity in reviews can sometimes be brought about by different focuses. For example, one review might recommend acceptance because of the novelty of the topic while another might recommend rejection because of flaws in the data collection or analysis. In such cases, the discussion might focus on which criterion matters more for ITiCSE.
In the end, if there are differences that cannot be resolved, the reviews should reflect that. We are not asking reviewers to present a unified front at all costs; we are asking you to more or less agree about a paper if that is possible.
Discussion takes place by way of comments. When you are viewing the reviews and comments for a paper, you add a comment by using the Add comment field at the bottom of the screen. The reviewers and APCs are all mutually anonymous, so please do not divulge your identity in a comment. However, to let the other participants know which reviewer is saying what, please identify yourself and one another by review number. For example, you might write "R4 here: APC, I see your point, but I still think R3 makes a very good case that the required changes would make the paper too different from the one that was submitted for review." You will appreciate that there is little point in writing simply "I agree" when different people have expressed opposing thoughts. Which one are you, and which one are you agreeing with?
Comments on a paper are seen only by the other reviewers for that paper, the APC for that paper, and the program chairs. In particular, they are not seen by the paper's authors. For this reason, if a point arises during the discussion that would benefit the authors, you should revise your review to include it. Likewise, if in the discussion you change your mind about points you have made, or about your overall impression of the paper, you must submit a revised review that expresses your new viewpoint. However, please ensure that your review stands in its own right; it is not appropriate to make a point by referring the authors to another review.
When revising a review, please ensure that the written comments and the numerical scores still match.
APCs, by about halfway through the discussion period you should have a good idea about your overall recommendation for each paper. Your metareview will consist of an accept/reject recommendation, a text field that will be seen by the paper's authors, and a confidential text field that in most cases will not be needed. If the reviews for a paper are in reasonable agreement, your metareview might simply note this. If there is some disagreement, your metareview might refresh the important points and explain the reasons for your recommendation. If you post your metareview before the end of the discussion period, the reviewers can comment on it, too, if they wish, and you will have the opportunity to revise it.
Once the discussion and metareview period is over, the program chairs will make the decision for each paper. Please remember that they must consider many factors when making these decisions, and that they might not always make the decision recommended by the APC. Also, the program chairs might wish to discuss particular papers with individual APCs.
Please be aware that we will be looking informally at the performance of all reviewers. Factors considered will be the quality of their initial reviews, their participation in the discussion, and their willingness to change their position on a paper if it seems appropriate.
Optional: Saturday 17 March - Friday 23 March: reviewing submissions for posters and for tips, techniques, and courseware
In the week beginning 17 March we will require reviewers for posters and for tips, techniques, and courseware. There will be far fewer of these submissions, they will be shorter than the full paper submissions, and they will generally be much easier to review. When we know how many of these submissions there are, we will call for volunteers to undertake this additional reviewing.
These reviews will have no APCs and no discussion period; they will be considered directly by the chairs for their respective tracks.