3.1.3. Phase 3: Validation
In this third phase, we presented the set of indicators and visualizations to an audience of Wikimedians. In total, we participated in the four conferences we mentioned in the previous section, where we were allocated from 15 minutes to 1 hour for presentations. These were held between August and November 2021. Each conference session had time allocated for questions and two of them led to focus groups sessions which we facilitated following a focus group protocol [41,42]. Two additional sessions were organized for the Volunteer support network, a dedicated group of people interested in supporting Wikimedia volunteers in different communities (Meta contributors, ’Connect/Volunteer supporters network’, Meta, discussion about Wikimedia projects, 18 November 2019, 21:20 UTC, https://meta.wikimedia.org/w/index.php?title=Connect/Volunteer_supporters_network&oldid=19568945 [accessed 19 February 2022]).
These sessions are useful for getting people’s perceptions and attitudes about any particular concept. In each group discussion and focus group, rather than engaging in discussion, we preferred setting the debate and then reminding that the indicators were a work in progress and that their input would be very valuable. We specifically asked community members four different things:
- To discuss the usefulness of each metric;
- To give any suggestion to improve the analysis (e.g., setting a different target value, or modifying some aspect of the data visualization);
- To consider the actor or actors who can take more responsibility in improving this Vital Sign for your Wikipedia language edition. Options are the affiliate, Wikimedia Foundation, and specific groups of editors;
- To explain which actions could help improve the current situation in relation to the target in the short or midterm horizon.
From each of the sessions, notes were taken and stored for interpretation among the researchers. Some comments were literally transcribed, while some ideas were written down in order to improve on the next iteration of metric design. During the interpretation of the notes, words, context, and duration of each part of the discussion were taken into account in order to assess their value.
3.2. Definition of the Indicators
We propose the definition of 6 indicators that we call “Vital Signs” to reach objective [O2] of exploring the degree of renewal within communities. In the following, we refer to each vital sign as an indicator, although in some cases, a vital sign may be composed of more than one indicator. In medicine, vital signs indicate the status of the body’s vital, life-sustaining functions. These measurements are taken to help assess the general physical health of a person, give clues to possible diseases, and show progress toward recovery.
In the case of Wikipedia, Vital Signs are related to the community’s capacity and function to grow or renew itself. Three of them are focused on the entire group of active editors creating content: retention, stability, and balance; the other four are related to more specific community functions: admins, specialists, and global community participation. We believe that obtaining values for the capabilities of the current active community of editors in these areas can constitute a reference point to plan to guarantee transparency and openness in these areas, to observe growth and renewal, and at the same time, to foresee and prevent future risks.
Based on our interpretation of the data for a wide array of Wikipedia language editions, we estimated a set of target values for each of the Vital Signs of what it might constitute a healthy, renewing community with growth potential. We think that these targets are reasonable for established communities (more than 50 active editors per month) who have achieved a critical mass that allows them to be sustainable in all the necessary tasks to develop a Wikipedia language edition. As these target values may be more easily interpreted and understood in comparison to the current values, we show them in Section 4, along with the results obtained for the selected analyzed communities.
