Table 1 Current GRRAS Guidelines and Suggested Additions to GRRAS

TITLE AND ABSTRACT

1. Identify in title or abstract that interrater/intrarater reliability or agreement was investigated.

INTRODUCTION

2. Name and describe the diagnostic or measurement device of interest explicitly.

3. Specify the subject population of interest.

Describe the database used to select the cases and the quality of that data.

4. Specify the rater population of interest (if applicable).

5. Describe what is already known about reliability and agreement and provide a rationale for the study (if applicable).

METHODS

6. Explain how the sample size was chosen. State the determined number of raters, subjects/objects, and replicate observations.

Describe the sampling method and the underlying population of both subjects and raters.

7. Define the Reference Standard diagnosis.

8. Describe the sampling method.

9. Describe the measurement/rating process (e.g. time interval between repeated measurements, availability of clinical information, blinding).

10. State whether measurements/ratings were conducted independently.

11. Describe the statistical analysis.

RESULTS

12. State the actual number of raters and subjects/objects which were included and the number of replicate observations which were conducted.

13. Describe the sample characteristics of raters and subjects (e.g. training, experience).

14. Report estimates of reliability and agreement including measures of statistical uncertainty.

DISCUSSION

15. Discuss the practical relevance of results.

AUXILIARY MATERIAL

16. Provide detailed results if possible (e.g. online).