A Simplified and Scalable Approach to Measuring Rater Consistency in Large-Scale Assessments
🧠 About the Index
The Composite Interrater Reliability Index (CIRI) is a new reliability metric developed to address the challenges of measuring interrater consistency in large-scale assessment environments, such as Intensive English Programs with dozens of active raters.
Developed by our founder, Dr. Alper Şahin, CIRI provides a practical, automated, and transparent method for evaluating how consistently raters score student performances across writing and speaking tasks
Traditional indices like Cohen’s Kappa, Pearson correlation, or Many-Facet Rasch Model often become computationally complex, time-consuming, and impractical when applied to high-volume rater data.
CIRI eliminates these limitations by focusing on total performance scores rather than individual rubric subscores, enabling faster and more meaningful consistency checks without requiring specialized software
⚙️ How It Works
CIRI uses tolerance-based intervals to determine how closely raters’ total scores align with an Estimated True Score (TE) derived from expert evaluations.
The index calculates the proportion of raters whose scores fall within ±2.5%, ±5%, ±7.5%, and ±10% of the maximum obtainable score and then averages these results to yield the CIRI Overall value.
CIRI’s structure allows administrators to interpret reliability levels quickly and clearly — a crucial advantage in operational testing.
🌍 Research and Findings
CIRI was piloted in a large-scale study involving 92 raters evaluating writing and speaking performances using Asymmetric Rubrics (Şahin, 2024).
Results demonstrated that most assessment tasks yielded average to good interrater reliability, with CIRI values ranging between .38 and .48, and one writing task reaching .62.
These findings confirm CIRI’s ability to capture meaningful variability in rater consistency while maintaining simplicity, transparency, and scalability
CIRI is particularly effective for high-volume rating systems, providing a composite yet interpretable measure of consistency that can be easily integrated into institutional quality assurance systems.
💼 Access and Consultation
The Composite Interrater Reliability Index (CIRI) is currently available for free for academic and research use. Institutions and researchers interested in implementing CIRI or receiving training and consultancy on rater consistency analysis using CIRI can contact: 📧 info@asimetry.com
📘 Publication and Future Work
CIRI’s methodological details and pilot study results were presented at the 9th International Conference on Measurement and Evaluation in Education and Psychology (AMEEP) in 2025 and are part of an upcoming peer-reviewed publication. Abstract of this presentation can be found below.
The index is being further refined for integration with digital assessment platforms and automated reliability dashboards under ASiMETRY’s research and development initiatives.
📄 Citation
Şahin, A. (2025, September 3–6). Composite Interrater Reliability Index (CIRI): A new index for consistency in high-volume rater environments [Paper presentation]. 9th International Conference on Measurement and Evaluation in Education and Psychology (AMEEP), Eskişehir, Türkiye.
