not provided a definitive answer to this question, but it has demonstrated that the approach of providing a clear and specific framework can be highly productive. We encountered no evidence, at any rate, that reviewers felt constrained by the criteria provided for the review process. They were clearly capable of introducing their own criteria into the qualitative discussion of articles as appropriate, such as in the following:
- "The discussion of the Monologion is fairly good. There are some factual errors or at least infelicities: 'monologium' and 'proslogium' should be 'monologion' and 'proslogion'. Anselm was most likely canonised in 1494. The treatments of the proslogion and cur deus homo are superficial." (Reviewer 2 – St Anselm)
We definitely believe that the comparative approach worked very effectively, and we would certainly recommend using that more widely, all other considerations being equal. This was clearly demonstrated on a number of occasions by comments such as the following:
- "Reading the second article made me realise how poor the first article was in that it did not cover the subject comprehensively and focused excessively on one aspect that could be viewed as peripheral. In the second article, the structure and the reference to sources were ideal." (Reviewer 1 – antibiotic resistance)
- "The most important differences separating the two articles were conciseness and scope of information." (Reviewer 2 – Mutation)
- "Article 1 is superior in almost all respects to article 2. The difference is particularly striking in terms of structure, references, factual accuracy, grammar and language." (Reviewer 2 – Numero Racional)
In addition to the way in which comparing articles managed to focus reviewers' awareness of the qualities and weaknesses of specific articles, it was interesting to see on a number of occasions the way that comparison generated insights into the ways in which an article on a particular topic might usefully combine the insights and approach of multiple articles:
- "Although both articles address the same basic issue of global warming, each of them with very different views, I find both very instructive and entertaining. The first privileges the sociological items, while the second starts with a more geophysical outlook. I find them both very good and complementary." (Reviewer 1 – Cambio Climatico)
- "I preferred the first article to the second. It is written in a more scholarly manner and it provides a lot of references. I found the second paper still a draft, and this might be the case. Ideally you would combine the two to give a more comprehensive picture of preschool education." (Reviewer 2 – Preschool Education)
One methodological issue that raises perplexing questions is the fact that reviewers' judgments were not always in exact agreement with one another on specific articles. To some extent, such differences reflect the previous point about the different perspectives on particular topics that emerge from different articles. We do not consider, certainly, that different viewpoints on the same topic are necessarily invalid – indeed, they are part and parcel of academic life, as is variation in academics' judgment of quality. Just as an article submitted to a journal for peer review will very often receive diverging judgments, so did a number of the online encyclopaedia articles in the present sample (such as, for instance, the following where one reviewer differed markedly from the other reviewer or reviewers:
52
