Research talk:Voice and exit in a voluntary work environment/Team work effect:a viability test

Feedback request

@Iadmc: @Stuartyeates: @Jane023: @Rosiestep: @SPatton (WMF): @Rich Farmbrough: @Neotarf: Hi all. You've all communicated with us about this research earlier in 2017 at Research talk:Voice and exit in a voluntary work environment. I'm following up with you as we have made some good progress on a proposed framework we would like to do some experiments with, to assess the viability of the direction we're moving towards. We have documented the suggested framework at Research:Voice_and_exit_in_a_voluntary_work_environment/Team_work_effect:a_viability_test. If you are interested, please read it and share your feedback with us.

To give you a sense in terms of the next steps and timelines: We will be requesting feedback from the people who may be interested in this research in the coming 2 weeks. We will then review the feedback and incorporate it as much as possible in the process. If we can't incorporate, hopefully we will convince you and others why that is the case. Then, we spend another 2 weeks finalizing the decision on which communities may be good candidates for this first version of the experiment. (We don't want to exhaust many communities, since this experiment will serve as a proof of concept that the direction is correct, or not.)

I'm hoping that we arrive at a place where we can run this tets in early April 2018 and have the results before the end of June 2018 (hopefully earlier, but this kind of experiments may need longer times as we rely on people being willing to participate in them.)

Anything you want to share, we're looking forward to hear. --LZia (WMF) (talk) 19:51, 2 February 2018 (UTC)

Feedback

1. The survey 'Doctoral Degree' should be 'Doctoral Degree (Phd)' to make a clear distinction in the case of medical doctors.
2. I would encourage you to match groups of three, rather than two, since there is much more opportunity for for positive group dynamics in a group of three
3. The editor community has historically been quite adamant they they should be kept informed. A page outlining the project should be created once all the details are settled and all emails should contain a link to it as 'further information'

cheers Stuartyeates (talk) 22:31, 2 February 2018 (UTC)

Stuartyeates Thanks for the feedback. Responses are below:

1. Done
2. There is one main issue I can see with teaming people up in groups of 3 and that is, for this specific iteration, we're talking about a relatively small scale experiment and finding matches in groups of 3 can become really hard. It's more likely to find 2 people interested in a topic than 3. Once we have a clearer baseline for groups of 2 and we have a solid pipeline we can run experiments in, we should definitely consider groups of 3. For example, it would be great to assess the potential trade-off between group dynamics and coordination failure/complexities in groups of 3 versus 2.
3. This is a hard one. :) There are a few things we absolutely do:

• Have a privacy statement for the experiment/emails/surveys always linked from the emails and survey where the user can learn how the data that is going to be collected will be used.
• Have a clear description of the project for the more experienced Wikimedia community.

We know that we cannot tell the participants details about the research as them knowing that we are, for example, testing for group effect can have impact on their responses. We basically lose the ability to be able to say what we learned from the study. Suppose this were to run on enwiki, what level of information would you want to see provided to the user knowing this complexity?

Thanks! :) --LZia (WMF) (talk) 19:12, 6 February 2018 (UTC)

teaming up

Hey LZia, thanks for putting this up and asking for input. I was pointed here by someone who came across, and it looks like a promising direction. I just was curious about one thing. You seem to assume at some point that the participant is aware that you'll be trying to team them up later. However, it does not become clear to me at which point you inform them of that intention. Or am I overlooking something? Was this supposed to be part of the welcoming email, or is there a message before that? I can imagine it's a tricky balance between using it to encourage them to participate in the survey, transparency and not influencing them too much before the survey. Effeietsanders (talk) 08:41, 3 February 2018 (UTC)

Hi Effeietsanders. You're right. The answer as to where we reveal this information depends on the research questions as you know. For this specific test, one of the things we want to assess is whether newcomers would engage with a survey, and if so, whether they will respond to the questions. We intentionally don't want to create an incentive (or disincentive) for the person to participate in the survey. If we see this direction is a promising direction, then we may want to consider bringing that information to the first email. Does this make sense to you? --LZia (WMF) (talk) 19:12, 6 February 2018 (UTC)

Some feedback

I think the survey asked too many/too detailed questions of a new editor. I think some people (women?) may be inclined to ignore it because they don't want to divulge anything about themselves. As for those who complete the survey, it's too bad that only one-third will actually get paired with someone. I wonder what the reaction(s) might be of those who don't get paired? --Rosiestep (talk) 01:59, 4 February 2018 (UTC)