News and Events

3rd Intercoder Reliability Workshop in Content Analysis

On Friday 6 July, the 3rd Intercoder Reliability workshop in Content Analysis will take place at the Tilburg University Department of Communication and Cognition. This edition is all about “unitizing”.

We kindly invite you to come as a presenter or a visitor. We aim at participants from social sciences and humanities who have experience with quantitative corpus analyses but are not yet advanced in unitizing. We have room for 25 participants (first come, first served).

Content analysis is an important method in the social sciences and humanities. However, it can be very difficult to achieve a satisfactory level of intercoder agreement. In most studies, annotating consists of coding predefined items, where coders only have to choose a category for each item. However, when the data is a continuum (e.g., text, audio, video), coders also have to choose the relevant parts of the continuum (units) before they categorize them. This is called unitizing.

It is difficult to get a good agreement value for coding of predefined items, in particular when variables are subjective (e.g., metaphor types, coherence relations, informal language, filmic narratives). But it is even more difficult in the case of unitizing since there are additional disagreements possible concerning the position and the presence of units. Correspondingly, methods to assess agreement for unitizing are much more difficult to elaborate than the ones for predefined items (such as the famous Cohen’s kappa) because there are two types of discrepancies (position and category) that interfere.

The 3rd Intercoder Reliability (ICR) Workshop is about this complex phenomenon of unitizing. What does unitizing entail? How to assess agreement? Which problems do researchers have? To what extent are these problems subject-dependent? What solutions are possible? These and similar questions will be addressed in this workshop.


The keynote speaker of the workshop is Yann Mathet, researcher at Université de Caen and the research group Human Language Technology in the GREYC research lab in digital sciences. Mathet’s research is focused on unitizing in the fields of Computational Linguistics and Natural Language Processing. One of his leading publications is called ‘On the reliability of unitizing textual continua: Further developments’ for which he collaborated with (inter alia) Klaus Krippendorff.

Set up

Yann Mathet will first give a lecture on unitizing, followed by a hands-on workshop in the afternoon. In between, participants can give a (poster) presentation about their own experiences with unitizing, emphasizing difficulties they cope with in their research. It is not needed to present solutions per se, we strongly invite participants to describe their problems as input for a small-scaled group discussion. We aim at participants who have experience with quantitative corpus analyses but are not yet advanced in unitizing. The workshop is limited to 25 participants.


We invite scholars from the social sciences and humanities to participate in this workshop:

  • Scholars can give an oral presentation: please email your abstract.
  • Scholars can prepare a poster presentation: please email your abstract.
  • Scholars can attend as visitor: please enroll via email.

Guidelines for abstracts

Scholars who would like to give an oral or poster presentation can submit an abstract of 300 words. The duration of an oral presentation is 30 minutes (approximately 15 minutes for presentation and 15 minutes for discussion).

If you want to submit an abstract, please structure it as follows:

  • Title, author(s), affiliation(s), contact email and phone number for first author
  • Abstract
  • Keywords (3-5)
  • Preferred category (oral / poster presentation)

Please note that if your presentation is selected for inclusion in the workshop program, the organizers will do their best to accommodate your preferred choice. However, the final category will ultimately depend on a number of factors, including the availability of spaces and time slots in the schedule.

Important dates

  • Deadline for submission abstracts for presentations: May 14, 2018
  • Notification presentations: May 22, 2018
  • Deadline for enrollment participants without presentation: 15 June
  • ICR 2018: July 6, 2018

Abstracts and enrollment notifications can be sent to Renske van Enschot.

Topics, approaches and modes

ICR 2018 solicits papers on any topic or approach in any mode (verbal, visual, audio, video, multimodal, etc.) related to intercoder reliability and unitizing. General topics and approaches of interest include, but are not limited to:

  • Affect/emotion and sentiment
  • Coherence relations
  • Conversational human voice
  • Grammar errors and youth language
  • Language intensity and hyperboles
  • Negotiation strategies
  • Nonverbal behavior
  • Multimedia and multimodality
  • Sarcasm and irony
  • Metaphors
  • Argumentation
  • Natural Language Processing
  • Computational linguistics
  • Conversation analysis
  • Interview analysis
  • Storytelling and narratives

About us

ICR 2018 is organized by the Department of Communication and Cognition (DCC) of the Tilburg School of Humanities and Digital Sciences (TSHD) at Tilburg University. The organizing committee consists of Renske van Enschot and Christine Liebrecht.
Prior ICR Workshops were organized by Radboud University Nijmegen (2014) and VU University Amsterdam (2015).

When: 06 July 2018 09:30

End date: 06 July 2018 18:00