Grants talk:IdeaLab/Measure replacement rate among the admins

Individual engagement grant?

I only wanted to type down the idea as part of the Inspire campaign… :D — Jeblad 22:27, 19 July 2018 (UTC)

Related Research

Those interested in this idea, may also be interested in this Research paper. Where they found that admins on Wikia wikis did not seems to turn over very much.

Link to a draft version of the article.

It was published in the Journal of Communication which is a very good social science journal. — The preceding unsigned comment was added by Groceryheist (talk)

Comments from PEarley (WMF)

Hey @Jeblad:, @Gryllida:, and @Mustafa desamangalam:,

This is a great idea! I know that communities often struggle to determine whether they have enough admins, and whether their processes are refreshing the admin group effectively. Some of your ideas about ratios have been looked at by Holder on de.wiki, here’s a presentation they gave at de.wiki’s 2015’s Adminconvention.

If you are interested in developing this proposal further in terms of funding, myself and some of my colleagues working on the Community Health initiative would be happy to look at some of the technical feasibility aspects of the idea. This might help the proposal get more specific in terms of work needed. Best regards, Patrick Earley (WMF) (talk) 21:02, 7 September 2018 (UTC)

@PEarley (WMF): It could be interesting to look into this. Email me?

The core idea isn't anything more than a spacial page with results from an analysis of number of logged group transitions over a year for an admin normalized over the number of logged group transitions over a year for an autoconfirmed user. They should be over a year to remove periodic changes. Unfortunately promotion to autoconfirmed isn't logged. (Anymore?) It is although possible to recreate this information.

Just as an additional note; there are several other measurements that can be done, and it isn't quite clear which one is the best. One measure is the mean time before promotion from one group to another group, typically from autoconfirmed to admin. If this number grows to big then the admin group becomes aged. Another measure is the number of admins normalized against the number of active users. If this number gets to low the admins will be overloaded. A better (at least simpler) measure of this could be to simply check the fraction of admins in the RC-feed. This number reflects the actual activity level, which is a better overall measure. — Jeblad 01:07, 8 September 2018 (UTC)

Redefinition, clarification

Mockup of the proposed special page for health metrics.

It seems like this is several slightly different metrics

The number of real changes in the group normalized over the total number of members in the group (internal change rate)

This is number the number of new members joining the group plus the number of old members leaving the group normalized over two times the total number of members in the group. Those numbers are straight out of the user rights log given the members of the group. This is perhaps the easiest metric.

Whether previous members should be counted as new members can be debated, but in my opinion they should not. They will not create a real influx of new members, they are only re-appointments of the same members. By counting over a long enough time period re-appointments can easily be canceled out. Still note that the time period must not be longer than the fixed term used at some of the projects.

This metric is probably the one most users would use as a metric for a stale group.

Note that internal change rate is easier to calculate and interpret than replacement rate. The later can only be calculated for incoming new members, as old members leaving the group is nearly impossible to calculate. When has a member truly left the group?

The number of real changes in the group normalized over the total number of active users (external change rate)

This is the number of new members joining the group plus the number of old members leaving the group normalized over two times the total number of active users or the log of the active users.

It is not clear whether the internal or external change rate should be used. The internal one is correct whether the project is oligarchic or not, while the external one should be normalized over the log of the active users if the project is oligarchic. (Ie to be able to predict correct numbers, although if the project is oligarchic it has serious problems.)

Note that this isn't a probability.

The mean time the members stay in the group (mean member time)

This the accumulated time for all members in the group, divided by the total number of members in the group, given the current active members. This metric comes from the user rights log.

This is a variant of the change rate as a low number for that will give a long time for mean member time. At some projects this number can approach the lifetime of the project.

The probability of one new group member in a lower group being promoted to an higher group (promote probability)

For most groups this is simply an extract from the user rights log. Unfortunately it is not possible to do this for the autoconfirmed group, so an alternate method to extract the new group members must be created. Also note that the influx to the higher group must be truly new members, and not just re-appointment of previous members (admins).

Usually this number is very low on Wikipedia, and because it is so low users have stopped expecting to become an admin. This leads to a higher status of being an admin, and people are then less likely to accept being wrong. Ie the social fall has become to large.

The mean time before promotion from one group to another group (mean time before promotion)

This number is simply a variant of promote probability but it is a lot easier to understand.

Because the promote probability is so extremely small the mean time before promotion grows so large it is nearly impossible for an ordinary user to be promoted during his or her lifetime on Wikipedia.

This number can be adjusted for the conditional probability that the mean time is for those that is in fact promoted. The number is still very high.

The normalized activity level for a group against the total activity in recent changes (activity level)

The activity level of a group can be measured and normalized against the overall activity level in the recent changes feed, thereby giving a measure that say whether there are enough members in the group. That is the activity in the recent changes feed is used as a proxy for the groups overall activity. It does not say anything about the group actually doing their job, but it does say whether the group is large enough compared to some limit.

This number can be calculated from activity in the recent changes feed, with some additional lookup.

Note that usually we will want a metric for the admin group, but the metrics does make sense for other groups too. Still it is the admin group that most likely turn oligarchy. — Jeblad 13:50, 8 September 2018 (UTC)

A few metrics from the paper of Shaw and Hill (that is the hypothesis’ about oligarchiness reformulated)

The number of new group members (aka administrators) over number of active contributors

This is H1; "The probability of adding new administrators declines as wikis’ contributor bases grow". The number should probably be normalized over active contributors and not number of registered users.

The number of contributions to project pages by group members (aka admins) over total number of contributions to project pages

This is H2; "Controlling for the total number of contributions to administrative pages, administrators will contribute more to administrative pages as wikis’ contributor bases grow".

The number of reverts by administrators of experienced contributors over the total number of contributions made by experienced contributors

This is H3; "Controlling for the total number of contributions made by experienced contributors, the number of reverts by administrators of such contributions will increase as wikis’ contributor bases grow".

These three indicators can be used as trends, comparing over an interval. — Jeblad 21:43, 9 September 2018 (UTC)