ITiCSE 2019 Paper Review Process

ITiCSE papers are reviewed using a double-blind process managed through EasyChair. The reviewing process has three phases: bidding, reviewing, and discussion and metareviewing

Reviewers are expected to provide high-quality reviews to provide authors with feedback that will help to improve the work for publication or for future submission. Associate program chairs (APCs) lead the discussion among reviewers and metareview each paper, providing a recommendation and feedback to the program chairs.

Each paper submission is expected to receive five reviews and a metareview.

All reviews are submitted through EasyChair. In EasyChair, reviewers are called ‘PC members’ and APCs are called ‘senior PC members’.

All paper submissions are anonymous. Reviewers and APCs are and should remain anonymous to one another. When writing comments on a review during discussion, please refer to yourself and other reviewers by the number of their review.

Reviewer and APC timeline

The following dates describe the timeline for reviewer & APC work on ITiCSE 2019. Note that the review period is just two weeks; this is to ensure that both the review period and the discussion period are complete before people need to travel to SIGCSE.

All reviewers and APCs are expected to engage with the process throughout the five weeks from Monday 21 January to Friday 22 February, as described below. Please consider your workload and other commitments around these dates before offering your services as a reviewer or APC. Those who cannot commit to engagement throughout this period should not volunteer to be reviewers or APCs for ITiCSE 2019.

ActivityStart dateEnd date
(Anywhere on Earth, UTC-12)
Bidding (1 week)Mon 21 JanuarySun 27 January
Reviewing (2 weeks)Wed 30 JanuaryTue 12 February
Discussion and metareviewing (10 days)Wed 13 FebruaryFri 22 February

Expected workload

  • Reviewers: 4-6 full papers (each up to 6 pages + references)
  • APCs: 8-12 full papers (each up to 6 pages + references)

These estimates are based on the workloads over the past few years. Of course they are subject to the number of submissions, so actual workloads for ITiCSE 2019 might be a little higher or lower.

The program chairs will not generally not accommodate requests for lighter review loads. If you are not able to commit to the full load, please do not volunteer your services.


Reviewers and APCs will bid for papers that they are interested in reviewing or metareviewing during the week between abstract submission and full paper submission. If reviewers take the bidding process seriously, this reduces the chance of weak reviews arising from a lack of familiarity with the content of a submission. As part of the bidding process, reviewers will be required to note papers with which they have a conflict of interest (papers that they recognise are from colleagues or project partners).

It is important to understand that bidding does not mean choosing the papers you will review. That choice will be made by EasyChair, using an algorithm that tries to allocate the papers reasonably fairly. Ideally, bidding is a process of identifying all of the papers that you would like to review or are willing to review. The more papers you bid for, the more likely you are to be allocated papers from your list.

Bidding on just a small number of papers influences the algorithm unfairly in your favour. To counter this, anyone who bids on too few papers will have their bids supplemented with an arbitrary selection of papers. If you do not wish this to happen to you, please ensure that you bid ‘yes’ or ‘maybe’ for at least 30 papers, with at least 15 of those being ‘yes’.

Once you are notified that bidding is open, you should log in to your EasyChair account and access the ITiCSE 2019 conference with the appropriate role (‘PC member’ or ‘senior PC member’).

You are likely to find yourself on a page listing the Submissions to the conference, but this is not the page you want. Instead click on the Paper Bidding menu. Again you will see a list of the submissions, but for each one you can now click links to indicate the level of your interest in reviewing them. To see the abstracts as well as the titles, click either Show full abstracts or Show abstract summaries at the upper right of the screen.

There is no need to click No for papers you do not wish to review; No is the default option. You should click Conflict for a paper if you believe you know who wrote it, and thus cannot be sure of reviewing it impartially.

The papers are presented in a random order for each bidder, to help ensure a good spread of bids. However, this order is preserved for each bidder, so if you spread your bidding over several sessions, the papers will appear in the same order each time you access them.

The bids are colour-coded, but the selected yes/maybe/no/conflict link is also indicated in bold face, which will help those who have trouble distinguishing the colours.


Reviewer responsibilities

A few days after the close of bidding, reviewers and APCs will be notified which papers they have been assigned, and paper reviewing will begin.

Reviewers are urged to download their assigned papers immediately and take a first look at them. If, for example, you discover that you cannot review a paper because you have discovered a conflict of interest, we would far rather be informed in the first few days of reviewing than on the last day.

When you log in to the appropriate role on EasyChair you will see a Reviews menu. On this page you can download the papers assigned to you. You should also click Add Review for one of the papers, to get a first look at the review form – but don’t submit the review until you’ve written it!

Please aim to complete your reviews before the deadline; it is a deadline, not a target submission time. When reviewers leave the task till the last minute, things can go wrong (illness, work emergency, family emergency, reviewing taking longer than you anticipated), and then the reviews are not done by the deadline – which means that we are left asking for volunteers to review extra papers at very short notice.

There will be periodic reminders during the review period. If at any time during this period you become aware that you will not have your reviews written by the deadline, please contact us immediately so that we can decide how to deal with it.

Once you have submitted your review for a paper, you will be able to see the other reviews already submitted for that paper, and EasyChair will notify you by email when further reviews are submitted. Please consider these other reviews, in preparation for the discussion period.

Here are some points that you should consider when reviewing.

1.      Your review should be written by you; please do not farm your reviews out to other people. If you believe that other people should be reviewing for ITiCSE, please encourage them to do so in their own right. If you do not have the time or the inclination to do your own reviewing for ITiCSE, please do not volunteer to do so.

2.      Your review should indicate whether the paper as submitted is acceptable for ITiCSE. There is no review-revise-review cycle, so papers should be marked as accept if they are currently acceptable, not if they have the potential to become acceptable after substantial revision. In particular, ITiCSE does not have a conditional accept. If a paper is not suitable for acceptance subject to minor revisions, you should recommend that it be rejected.

3.      Whether or not a paper is accepted, the authors will want to know why. Please ensure that your review gives reasons, explanations, examples of the points you make. Reviews just a few lines long are generally not helpful to the authors, the APCs, or the program chairs. On reading the text parts of the reviews, authors should understand why the paper was accepted or rejected.

4.      Be sure that your overall recommendation tallies with your impression of the paper. If you do not believe that the paper should be accepted, please do not give it a 4 just to be nice, or to err on the side of generosity. It is more helpful to the program chairs, the APCs, and the authors if your review is honest and transparent and your overall recommendation reflects your impression.

5.      Please do not include your preference for acceptance or rejection of a paper in the text part of your review. For example, do not write ‘this is why I am recommending that this paper be accepted’. Instead, use the provided radio buttons to make a recommendation based on your summary review.

6.      ITiCSE reviewing is double-blind. You should not know who wrote the papers that you review, and the authors should not know who reviewed it. Please therefore take care not to divulge your identity in your review, for example by saying ‘This paper that I wrote might be helpful to you.’

7.      Please appreciate that authors sometimes have to omit or disguise references to their own work in order for their paper to be anonymous. As a reviewer, you may give the authors the benefit of the doubt. Use the ‘Confidential comments to the committee’ box to indicate this to the program chairs; for example “This paper should reference Matt Jadud’s work, unless those references were removed for anonymity”.

8.      If you find that a paper you are reviewing is not completely anonymous, you may point this out to the authors. However, please do not divulge the authors’ identity to the other reviewers or the APC, and please continue to review the paper as though it were anonymous. If you feel that you cannot not review it objectively, please notify the program chairs immediately.

9.      Most papers have some flaws in their writing. The reviewers are not required to be editors, pointing out every flaw that requires correction. However, some reviewers choose to do this, and many authors appear to appreciate the additional service.

10.      It is a good idea for reviews to be grammatically correct – especially if they criticise the grammar of the paper. Just as a poorly written paper gives the reader a poor first impression, a poorly written review can make the authors less inclined to accept its content. Please read your own reviews carefully, and, if necessary, submit revised versions with the spelling and grammar errors corrected. If you choose to draw attention to spelling issues in a paper, please be aware that many words have different spellings in different parts of the world. Neither ‘analyze’ nor ‘analyse’ is wrong: they are different acceptable spellings; likewise ‘program’ and ‘programme’, ‘behavior’ and ‘behaviour’, and many more such pairs.

11.    When reviewing a paper, please be aware of the category in which it was submitted. There are three categories: computing education research; experience reports and tools; and new curricula, programs, degrees, and position papers. Different guidelines apply to each category, and it would be unreasonable, for example, to review an experience report as though it were a research paper. See Paper category guidelines for the guidelines for each category.

12.    Good papers will be relevant to an international audience. ITiCSE certainly accepts papers based on work from specific countries, but such papers are more valuable if they have clear lessons for people from other countries. For example, a paper that writes about making it easier for people from the mountain regions to succeed in a computing degree would be better received if it made a clear generalisation to other underrepresented minorities – or perhaps to people with better lung capacity – regardless of country.

13.    The limit of six pages (with references possibly extending beyond the sixth page) is a maximum, not a minimum. There is nothing wrong with shorter papers, if they are good and pertinent; they do not need to be padded out to six pages. At the same time, if a shorter paper clearly lacks something that would have strengthened it, feel free to point that out to the authors.

14.    It is acceptable to point out formatting issues, but not to make them the deciding factor in a recommendation to reject. Some authors are still struggling to come to grips with the new ACM templates. If in doubt, leave it to the proceedings chair to decide on a paper’s compatibility with the template.

15. Please be aware that different versions of the ACM template have different placeholders for the title and the authors’ names. There is nothing insidious about a paper that says it is being submitted to WOODSTOCK’18, June, 2018, El Paso, Texas USA, or that its authors are F. Surname et al. or G. Gubbiotti et al.; these are all defaults in different versions of the template. As with other aspects of the formatting, assume that it will be fixed if and when the paper is accepted for publication and revised.

APC responsibilities

Each submission will be assigned to one APC. As an APC, please carefully read each submission at the start of the review period so that you will be able to relate the reviews to it as they are submitted. You are expected to watch the reviews for your assigned papers as they come in; to do this you will need to explicitly add them to your watchlist so that you will receive email notifications. From the ITiCSE 2019 menu choose My watchlist, choose the Select all option, then Save watchlist. Please keep an eye open for obvious problems: a review might have been submitted for the wrong paper, or might identify the author or the reviewer, or might be so brief as to be unhelpful. We will do our best to notice such problems, but your attention will also help.

Emergency reviewing (optional)

Despite the best of intentions, some reviewers find that they are not able to review the papers assigned to them and others simply drop out of contact for various reasons. The program chairs often discover this only at the end of the reviewing period. They then need to find reviewers who have completed their assigned reviews and are able to review one or two more papers within a day or two.

The call for emergency reviewers generally goes to all reviewers, but there is no expectation that all reviewers will respond. We understand that most reviewers now have other calls on their time, and we are appreciative and grateful that a few are able to step in and take on this additional load.

Discussion and meta-reviewing

The discussion period facilitates communication among the reviewers and APCs to ensure that the APC makes the best recommendation to the program chairs and the submission is given full consideration in the review process.

Reviewer responsibilities during discussion and meta-review

As a reviewer, you are expected to engage with the discussion on each of your papers during the discussion period. Read the reviews from the other reviewers and take part in the discussion using the Comments feature in EasyChair (beneath all the reviews). If, after reading the other reviews and taking part in the discussion, you are inclined to revise your review, please do so. There is no obligation to revise your reviews, but compatible reviews from all reviewers make matters easier for the APCs, the program chairs, and the authors of the submission.

If you do revise your review, please do not refer explicitly to other reviews. It is not appropriate to write, for example, “I agree with review 5 about the paper’s shortcomings”. It should be possible to read each review independently of the others.

APC responsibilities during discussion and meta-review

As an APC, you will lead and encourage discussion among the reviewers, especially for submissions with divergent reviews. This discussion will help you to form an opinion on whether the paper should be accepted.

Ideally, the reviewers and the APC will come close to consensus on the quality of the submission. Different reviewers might see different strengths and weaknesses in the paper, or might weight those strengths and weaknesses differently. However, it would be good if all reviewers, after reading the other reviews and taking part in the discussion, have fairly similar opinions about the paper, and are willing to revise their reviews if necessary to reflect those opinions.

Reviewers should never feel coerced to change their reviews, scores, or viewpoints in this process.

The APC will write a metareview, summarising the reviews, adding their own opinion if appropriate, and providing a recommendation for the program chairs. If the reviews are divergent, it is important for the metareview to summarise both positions. This metareview will be sent to the authors along with the regular reviews.

The text of the metareview should not explicitly include the APC’s accept/reject recommendation; there is a separate field for that. As an APC, you will only see a few of the submitted papers; please understand that your recommendation will be carefully considered, but that the program chairs might choose the opposite outcome when considering the full set of submissions and working to put together a broad yet coherent program.

APCs should note that confidential comments in the metareview are visible to all of the paper’s reviewers as well as the program chairs. If you wish to communicate just with the program chairs, please do so by email.

Additionally, the program chairs will ask APCs for feedback on the quality of reviews, to help inform decisions about future invitations to review for ITiCSE.

Paper category guidelines

There are three categories for papers. Reviewers and APCs will generally be assigned papers across all three categories.

Authors are asked to choose the track that they feel best fits their submission. Review the submission using the guidelines for the track the submission is in; not the track you would prefer it to be in. However, feel free to note if you believe that it would be a better fit with a different category.

Computing education research category

Papers submitted to the computing education research category typically describe an empirical computing education project.

Computing education research papers should adhere to rigorous standards, describing hypotheses, methods, and results as is typical for research studies. These will normally focus on topics relevant to computing education with emphasis on educational goals and knowledge units or topics relevant to computing education with statistical rigour; methods or techniques in computing education; evaluation of pedagogical approaches; and studies of the many different populations that are engaged in computing education, including (but not limited to) students, instructors, and issues of gender, diversity, and underrepresentation. We welcome replication papers and papers that present null or negative results that meet the criteria below.

For a typical paper in this track, here are some key factors to include (as an author) and to look for (as a reviewer):

  1. Are there one or more clearly stated research questions? Since the rest of the paper will be organised around these, it’s often good to put them in the abstract and in the first section of the paper.
  2. Are the questions of interest to the ITiCSE audience?
  3. Related work in computing education
    • Is the relevant work in computing education included? If not, a good review will give references to missing material. It is not enough to write “The related work section is incomplete”.
    • Do the authors clearly describe the relationship between prior work and the current research questions? In what ways does the current project build on prior work, and how is it different?
  4. Related work in educational theory
    • Is the project based in educational theory? If not, should it be and what are some theories the authors should consider?
    • Is the theory described clearly, with appropriate citations?
    • Is the theory’s relationship to the current project clearly described?
  5. Is the data gathering described clearly enough that the reader could reproduce it? Some key information to include:
    • About the data: why this particular type of data is relevant to your research questions
    • About the participants: how many, what was their background (are they instructors, students, graduates, etc.); what if any formal coursework have they had in computing; how many were men and how many women; and any other factors that are relevant to the author’s project
    • About the people gathering the data: what is their relationship to the participants? For example, if the data were collected from students in a class, was the instructor one of the researchers?
    • About the data-gathering process: did the project use surveys, interviews, samples of student work, something different? If surveys or interviews, exactly what questions were asked?
    • Each paper will ideally be self-contained. If the authors are unable to convey all of the content within the page limit, they may provide a link to supplemental materials; but those materials should be blinded for review, just as the paper is.
  6. Is the data analysis process/methodology sufficiently described that the reader could reproduce it?
    • What methodology was used?
    • Is the methodology described, with an appropriate citation?
    • Is the implementation of the methodology described clearly? How many people were involved? What process was used to resolve any disagreements?
    • Is the analysis process/methodology appropriate for answering the research questions?
  7. Is the analysis methodology something new to computing education research that might itself be a contribution?
  8. Are the results of the analysis clearly summarised?
  9. Are the results thoroughly discussed, including:
    • Their relationship to the research questions
    • Their relationship to prior work
    • The implications of the results for teaching
    • The implications of the results for future research
  10. Is there a discussion of threats to validity?

Experience reports and tools category

Experience reports and tools papers should carefully describe a computing education intervention and its context, and provide a rich reflection on what worked, what didn’t, and why. This category accepts experience reports and descriptions of teaching techniques or pedagogical tools. All papers in this category should provide enough detail to enable others to adopt the new innovation.

For a typical paper in this category, here are some key factors to include (as an author) and to look for (as a reviewer):

  1. Are there one or more clearly stated goals in this paper? Since the rest of the paper will be organised around these, it’s often good to put them in the abstract and in the first section of the paper.
  2. Is the experience or tool of interest to the ITiCSE audience?
  3. Related work in computing education
    • Is the relevant work in computing education included? If not, a good review will give references to missing material. It is not enough to write “The related work section is incomplete”.
    • Do the authors clearly describe the relationship between prior work and the experience being reported? In what ways does the current project build on prior work, and how is it different?
  4. Are the observations and/or findings from the experience or the use of a tool clearly summarised?
  5. Are the findings thoroughly discussed, including:
    • Their relationship to prior work
    • The implications of the results for future use
    • The implications of the results for teaching
    • Information on how to adopt or adapt teaching techniques and/or pedagogical tools in other contexts or institutions.

New curricula, programs, and degrees, and position papers

Papers in this category should describe new curricula, programs, or degrees, the motivating context for the new initiative, what it took to put the initiative into place, what the impact has been, and suggestions for others wishing to adopt it. This category may also include position papers, which are meant to engender fruitful academic discussion by presenting a defensible opinion about a computing education topic, substantiated with evidence.

  1. Is the innovation clearly stated? Since the rest of the paper will be organised around this, it’s often good to put it in the abstract and in the first section of the paper.
    • Description of the problem or need being addressed.
  2. Is the curricular innovation or position paper of interest to the ITiCSE audience?
  3. Related work in computing education
    • What prior solutions to this problem exist?
    • Is the relevant work in computing education included? If not, a good review will give references to missing material. It is not enough to write “The related work section is incomplete”.
    • Do the authors clearly describe the relationship between prior work and the material presented in this paper? In what ways does the current project build on prior work, and how is it different?
  4. Examination for discussion
    • How is the curricular innovation or position paper addressing this problem or need?
    • How is the curricular innovation or position paper different from previous ideas?
  5. Future success indicators
    • How could the curricular innovation or proposed idea be assessed if adopted or implemented?
    • In what context can the curricular innovation or proposed idea be used (large research institutions, community colleges, high schools)?
    • How difficult would it be to adopt the curricular innovation or proposed idea? For example, what human and financial resources would be needed?