This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Roberts, C. Articles by McNamee, R. Search for Related Content PubMed PubMed Citation Articles by Roberts, C. Articles by McNamee, R. Social Bookmarking                         What's this?

Assessing the reliability of ordered categorical scales using kappa-type statistics

Division of Epidemiology and Health Sciences, Stopford Building, The University of Manchester, Manchester, UK, chris.roberts{at}manchester.ac.uk

Roseanne McNamee

Division of Epidemiology and Health Sciences, Stopford Building, The University of Manchester, Manchester, UK

Methods for the analysis of reliability of ordered categorical scales are discussed, focussing on the limitation of the single summary-weighted kappa coefficients. A symmetric matrix of kappa-type coefficients is suggested as an alternative. The method is proposed as being suitable for ordinal scale where there is no underlying continuum. Their application is illustrated using two data sets from reliability studies. If, instead, distances between categories can be specified, a weighted mean of the matrix terms can be used as a summary measure. This is equal to a weighted kappa coefficient with squared weights, provided distances between adjacent categories are equal. When a study design corresponds to a one-way random effects model, estimates of precision of kappa-type coefficient, including the coefficients described here, can be obtained using the delta-method, bootstrap resampling by subjects or jack-knifing by subjects. In the case of interobserver reliability studies, where there may be systematic differences between observers, the investigator may wish to generalise to a population of observers and subjects. In this case, jack-knifing by observer and subject is suggested. Empirical comparisons are made between standard error estimates based on the delta-method, on jack-knifing by subjects and a two-way jack-knife by subjects and observers. The results suggest that standard errors based on the delta-method or jack-knifing by subject alone may be overly precise.

Statistical Methods in Medical Research, Vol. 14, No. 5, 493-514 (2005)
DOI: 10.1191/0962280205sm413oa


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				K. Steinsvag, M. Bratveit, B. E Moen, and H. Kromhout Inter-rater agreement in the assessment of exposure to carcinogens in the offshore petroleum industry Occup. Environ. Med., September 1, 2007; 64(9): 582 - 588. [Abstract] [Full Text] [PDF]