study. Of each pair of articles, one article was a Wikipedia entry on the topic and the other was an article on the same topic from the alternative online encyclopaedia for that language. Reviewers were not aware of the source of the articles and were asked to make no efforts to identify the same. All cues as to the source of the article were eliminated before the students viewed the article. This was carried out during the standardisation and anonymisation process, the details of which are described in section 3.4.2. Reviewers were asked to comment on the quality, accuracy, citability and style of each of the articles as well as on their opinions about the readability of the article and whether the information contained in it was, to the best of their knowledge, up to date. They were also asked to compare both articles within a pair, listing the strengths and limitations of each. Both quantitative and qualitative data were collected and reviewers were asked to confirm that they had made no attempt to identify the source of the articles by completing a declaration at the end of the review. The various dimensions assessed by the online feedback tool developed for the review process are detailed in Section 3.4.1.
3.3 Selection of articles
The selection of reviewers with strong academic credentials was considered to be paramount in this study, and therefore only after they had been recruited was it appropriate to seek articles that matched their areas of expertise sufficiently well.
A list of keywords for possible articles was drawn up based on the information provided by the students about:
- Their area of research and academic expertise.
- The nominated academic's area of research and academic expertise.
- Areas of overlap between the students' and academics' areas of research and expertise.
As it turned out, it was not always possible to select articles that mapped the students' and academics' areas of expertise exactly, as articles for these niche areas were not found to exist in many encyclopaedias or were found to be incomplete or of inadequate length. A second phase was then embarked on by the research team to select articles of substantial length (≥1.5 pages) that appeared most complete and comprehensive. This resulted in a list of possible articles that was much broader and less specialist than initially sought, and which did not map on to the niche aspects of the academic's expertise. Thus the selection of articles was constrained by two important factors: one, the need to find topics appropriate for the academics whom we were able to recruit to the project; secondly, that articles from different online encyclopaedias were of comparable substance and focus. (Such factors would need to be taken carefully into account when embarking on a future large-scale study, where the demands of finding large numbers of comparable articles are likely to be considerable.)
Nevertheless, the second phase allowed the compilation of the 22 pairs of articles for review, across three languages and four academic disciplines. The topics of the articles selected for review are listed in Table 3.2.
19
