their self-perceived qualification to comment. Wikipedia has continued to develop and refine its quality review processes in part as a result of the findings of the Nature study and of other similar studies. However, there has never been any attempt to replicate, better or extend Nature's study, across disciplines and languages. Such a study would not only allow a greater understanding of the accuracy and quality issues pertaining to Wikipedia entries but would also provide information on how such issues may be addressed and/ or resolved.
Recently, Wikipedia's founder Jimmy Wales highlighted the importance of such a task, i.e. a study inspired by the Nature study but employing greater rigour by carrying out the assessment of articles across languages and across a range of disciplines spanning the humanities and sciences, involving the following characteristics:
- Assessments carried out by academics and scholars.
- Assessments on each pair of articles carried out by multiple expert reviewers to establish inter-rater reliability and eliminate biases.
- Reviewers to be blind to the source of the article.
- A variety of constructs and dimensions relating to the quality, accuracy, style, references and overall judgment.
- Using both quantitative and qualitative rating techniques.
The importance of such a study would lie in the examination of articles in more than just the English language and in subjects other than solely science. This would allow differences in levels of accuracy and quality across languages and subject domains to be identified, which would inform decisions in the future, e.g. for editor recruitment efforts and the design of expert feedback mechanisms.
The size, scope and complexity of undertaking such a study would require considerable preliminary information on the methodology and design, compilation and functioning of rating scales, recruitment and location of the experts, and analysis and interpretation of results. As such it was decided that prior to the commencement of such a study, a small-scale preliminary project drawing on empirical evidence would be essential to determine a sound research methodology, which is the reason that the present study was undertaken.
This pilot study has therefore been carried out to collect and review preliminary evidence to inform the design of a larger, future study. The intention is that the results of this preliminary report will establish the best possible research approach, begin to hypothesise the best way for Wikipedia to measure and communicate the accuracy and quality of articles and provide a well-founded justification for seeking funding for a comprehensive study. This pilot study has been carried out for the Wikimedia Foundation by Epic, in partnership with the Department of Education at the University of Oxford, UK. The methodology, analysis and results of the study are presented in this report, followed by a discussion of the findings and the conclusion of the report.
12
