Executive Summary
1. Introduction
Previous studies, most notably the one carried out by the journal Nature in 2005, have sought to compare the quality of Wikipedia articles with that of similar articles in other online Encyclopaedias. In part as a result of the findings of such studies, Wikipedia has instigated a number of processes for assessing the quality of its entries, inviting readers and editors to rate articles according to criteria such as trustworthiness, neutrality, completeness and readability. Recently, Wikipedia's founder Jimmy Wales highlighted the value of conducting a study which analysed articles across both languages and subjects to allow differences in levels of accuracy and quality across language and subject domains to be identified. The results could inform editor recruitment efforts and the design of expert feedback mechanisms.
The size, scope and complexity of undertaking such a large-scale study necessitated gathering preliminary evidence to inform the methodology and design. It was therefore decided that a small-scale preliminary project would be essential to determine a sound research methodology, which is the reason that the present pilot study was undertaken. The present study, funded by the Wikimedia Foundation, presents the background, methodology, results and findings of a preliminary pilot conducted by Epic, a UK-based e-learning company, in partnership with the University of Oxford.
2. Aims and Objectives
The key aims of this pilot study are as follows:
- To explore the opinion of expert reviewers regarding attributes relating to the accuracy, quality and style of a sample of Wikipedia across a range of languages and disciplines.
- To compare the accuracy, quality, style, references and judgment of Wikipedia entries as rated by experts to analogous entries from popular online alternative encyclopaedias in the same language.
- To explore the viability of the methods used in respect of the first two aims for a possible future study on a larger scale.
3. Research Methodology
Three languages were selected for study: English, Spanish and Arabic. Pairs of articles in those languages were selected in the following broad disciplinary areas: (a) Humanities, (b) Social Sciences, (c) Mathematics, Physics and Life Sciences and (d) Medical Sciences. Each pair consisted of an article from Wikipedia, and an article from one of a range of comparator online encyclopaedias: Encyclopaedia Britannica (English), Enciclonet (Spanish), Mawsoah and Arab Encyclopaedia (Arabic).
Twenty-four postgraduate students of the University of Oxford were selected to help review pairs of articles and to identify academic experts in their fields who would be recruited to review the same pairs of articles. Thirty-three academic experts were finally recruited. All
5
