possessed doctorates and were employed in academic posts at a highly rated department within a well-established university. All students and academic experts were fluent in the target languages.
A feedback tool was devised for eliciting numerical scores and qualitative comments about the articles, which were reviewed blind by the academics, who were asked to certify that they had not sought out the original articles online during the review process. The feedback tool provided academics with a wide range of quality criteria, drawn from extensive previously published research.
Articles were standardised so as to erase information which helped to identify their origins; in particular, checks were carried out to ensure that a particular article was not the victim of vandalism (although this did not impact on article selection for the present study).
Twenty-two articles were selected in all. Some difficulty was encountered in finding articles of sufficient substance and scope in encyclopaedias paired with Wikipedia in different languages.
4. Data Coding and Analysis
Quantitative and qualitative data were analysed through separate processes. Quantitative data analysis was carried out on the sample overall, in relation to each language separately, and in relation to each disciplinary area separately. Data was coded in five main dimensions: i) accuracy, ii) references, iii) style/ readability, iv) overall judgment (including citability), v) overall quality score.
Qualitative analysis was initially carried out blind, and involved the reduction and display of reviewers' comments so that these could be compared with one another, in relation to specific articles, pairs of articles and across the sample as a whole. The qualitative analysis aimed to capture both the opinions of reviewers about specific aspects of the articles, and their overall judgments about each individually and in comparison with the other in the pair.
5. Results
All of the results outlined below are based on a small sample studied for the purposes of piloting the study's approach and methods, and these results cannot therefore be generalised to the wider output of the online encyclopaedias referred to.
Quantitative results for the articles reviewed show that the Wikipedia articles in this sample scored higher overall than the comparison articles with respect to accuracy, references, style/ readability and overall judgment. The scores for the latter item, which includes citability, indicated that none of the encyclopaedias were rated highly by academics in terms of suitability for citation in academic publications.
Results across languages showed that Wikipedia fared well in this sample against Encyclopaedia Britannica in terms of accuracy, references and overall judgement, but no better on style and overall quality score. The same was true of Enciclonet, but the Arabic encyclopaedias scored significantly higher on style than Wikipedia and equally well on the other criteria.
6
