1. Relevance: Is it paper highly relevant to COLING?
2. Readability/clarity: may be the paper well-structured and well-written?
3. Data/code supply: could be the data/code (as appropriate) open to the study community or perhaps is here a reason that is compelling why this is simply not possible?
4. Analysis: In the event that paper managed to reproduce the total link between the earlier in the day work, does it demonstrably set down exactly exactly what must be filled in to do therefore? It clearly identify what information was missing/the likely causes if it wasn’t able to replicate the results of earlier work, does?
5. Generalizability: Does the paper rise above replicating the outcomes regarding the initial to explore if they could be reproduced an additional environment? Alternatively, in instances of non-replicability, does the paper talk about the wider implications of this outcome?
6. Informativeness: To exactly what degree does the analysis reported within the paper deepen our comprehension of the methodology utilized or perhaps the nagging issue approached? Will the given information when you look at the paper assistance professionals using their range of technique/resource?
7. Significant contrast: along with pinpointing the experimental results being replicated, does the paper motivate why these specific email address details are a crucial target for reproduction and exactly just exactly what the long run implications are of the having been reproduced or been discovered to be non-reproducible?
8. General suggestion: there are numerous submissions that are good for slots at COLING 2018; essential could it be to feature this 1? Will people learn great deal by scanning this paper or seeing it presented? Please be decisive—it is much better to change from other reviewers than to grade every thing in the centre.

Site paper

Documents in this track provide a language resource that is new. This might be a corpus, but in addition could possibly be an annotation standard, device, an such like.

1. Relevance: Is this paper highly relevant to COLING? Will the resource presented likely be of good use to the community?
2. Readability/clarity: Through the means the paper is written, could you inform the way the resource ended up being produced, the way the quality of annotations (if any) ended up being examined, and exactly why the resource should always be of great interest?
3. Originality: Does the resource fill a need within the current number of available resources? Keep in mind that originality could possibly be when you look at the range of language/language genre or variety, within the design regarding the annotation scheme, within the scale regarding the resource, or nevertheless other parameters.
4. Site quality: what sort of quality control had been performed? If appropriate, had been inter-annotator contract calculated, if therefore, with appropriate metrics? Otherwise, what other assessment ended up being carried out, and just how acceptable were the outcome?
5. Site accessibility: might it be simple for researchers to down load or otherwise access the resource to be able to utilize it in their own personal work? As to the degree can perhaps work centered on this resource be provided? answers to incorporate: Yes, We have confirmed
6. Metadata: perform some writers explain whoever language use is captured into the resource and also to just what populations results that are experimental in the resource might be generalized to? In instance of annotated resources, will be the demographics regarding the annotators also characterized?
7. Significant contrast: could be the brand new resource situated with regards to existing work with the industry, including similar resources it took motivation from or improves on? Can it be clear what exactly is unique about the resource?
8. General suggestion: there are numerous submissions that are good for slots at COLING 2018; essential can it be to feature that one? Will people learn a complete lot by scanning this paper or seeing it presented? Please be decisive—it is way better to change from other reviewers rather than grade every thing at the center.

Position paper

A situation paper presents a challenge to old-fashioned reasoning or a futuristic brand new vision. It might open an area that is new unique technology, propose changes in existing research, or provide a fresh pair of ground guidelines.

1. Relevance: Is this paper highly relevant to COLING?
2. Readability/clarity: Is it clear exactly exactly what the positioning is the fact that paper is arguing for? Will be the arguments because of it laid down in an understandable means?
3. Soundness: Are the arguments presented within the paper coherent and relevant? Could be the eyesight well-defined, with success requirements? (Note: it must be possible to offer a top rating right here even although you don’t concur with the place taken because of the writers)
4. Creativity: How bold or novel may be the place used the paper? Does it express well-thought through and imaginative ground that is new?
5. Range: How much range for brand new scientific studies are opened up by this paper? Just exactly What effect could it have on existing areas and concerns?
6. Meaningful contrast: may be the paper well-situated pertaining to past work, both place documents essay help (taking exactly the same or opposing side on a single or comparable dilemmas) and appropriate theoretical or experimental work?
7. Substance: Does the paper have sufficient substance for a paper that is full-length? Could be the presssing problem sufficiently crucial? will be the arguments adequately varied and thoughtful?
8. General suggestion: there are lots of good submissions contending for slots at COLING 2018; essential will it be to feature this 1? Please be decisive—it is way better to vary from other reviewers rather than grade every thing at the center.

A study paper provides an organized summary of the literature up to now on a certain subject that assists the reader comprehend the kinds of concerns being inquired about the subject, the many approaches which were used, the way they connect with one another, and just what further research areas they open. A conference-length study paper should sufficiently be about a concentrated topic so it can try this effectively with into the web page limits.

1. Relevance: could be the paper highly relevant to COLING?
2. Readability/clarity: could be the paper generally speaking simple to follow and well organized?
3. Organization: Does the paper arrange the appropriate literature in a narrative and determine typical strands of inquiry?
4. Scope: Does the paper recognize a fairly focused area to review?
5. Thoroughness: because of the region identified to survey, does the paper address all the literature that is relevant? May be the literary works evaluated represented accurately?
6. Outlook: Does the paper recognize areas for future work and/or point out what clearly is certainly not yet managed inside the literary works surveyed?
7. Context: Does the paper situate research that is current within its historic context? (We don’t expect papers to begin with Pa?ini, yet during the exact same time one thing that just cites work from 2017 most likely does not capture just how current work pertains to greater photo.)