Research talk:Wikipedia Editors Survey 2012

May I inform the community?

Thanks for working on this new survey, Tilman. Is it too early to inform the enwp community about this survey? I'd like to give the enwp community time to provide feedback on the new surevey if it's OK with you. Thanks much. 64.40.54.212 05:15, 7 July 2012 (UTC)

Hi, of course this page is public and you don't need to ask for permission to draw attention to it if you feel that's useful ;) However I was hesitant to spam village pumps with it (and to avoid an enwiki bias, it would have been preferable to do that for other Wikipedias as well if one does it for the English Wikipedia); rather, I am in the process of pinging a few users that had already left feedback about the last version. What we need right now are just a few experienced Wikipedians going over the questions to spot possible problems. I might however post a request in the forum here on Meta. Regards, Tbayer (WMF) (talk) 03:06, 17 July 2012 (UTC)

"activities in the last 30 days"

Some of the replies to Q4a in the December 2011 survey

Percentages of "Often/Very often" replies in the last survey

Question 10 (formerly Q4a) ("... please indicate how often you have participated in the following activities in the last 30 days") generated some concern in the last issue of the survey. To address three points:

• Concern that for some activities which typically take place less often, a respondent may be have to answer "never" although they do engage in this activity comparatively frequently. In particular, someone who organizes six meetups per year might not have worked on this during the last 30 days and therefore would fall in the "never" column in the "I organize or help to organize events, workshops, meet-ups, or the annual Wikimania conferences" sub-question. - This can be assumed to "average out" over a large number of respondents, and indeed last time around 8% of 6,366 respondents replied at least "seldom" or more often to this sub-question, i.e. around 500 Wikipedians said that they had been involved in some form of event organizing in the last 30 days, which I find unlikely to be under-reporting. More likely, it could be over-reporting, relating to the next concern:
• "I suspect the resulting data will be somewhat distorted, a mix of answers strictly adhering to definition 'in the last 30 days' and others who don't." - The above example gives some substance to this concern, although the events sub-question may be an outlier. In my view, stating the concrete "30 days" timespan is useful to make it easier on respondents' memory (similarly to the "during the last seven days" restriction in the new question "How much time have you spent contributing"), and to provide a common frame of reference. If someone has been a Wikipedian for years, but only took up one particular activity two months ago, we don't want them to agonize whether that means "seldom" or not when answering this question. For this reason, I'm reluctant to drop the "30 days"; I'll check if there is a possibility to make them more visually prominent.
• "Seldom", "often" etc. are subjective terms: That's true, but asking respondents to pick absolute numbers instead (how many typos fixed/disputes resolved/MediaWiki lines coded last month?) doesn't seem very realistic, and the results from the last survey still appear meaningful, at least for the ten or so most frequently names activities (cf. blog post).

Still, suggestions on how to improve this question are welcome, although we need to finalize it soon and want to preserve consistency with the last surveys as much as possible. Regards, Tbayer (WMF) (talk) 03:06, 17 July 2012 (UTC)

In the second point above you say that the 30 days limit is good, because what would respondents think if they [...]. Well, why not handle the case explicitely? "If you have participated in an activity seldom in the past but often recently, or conversely, choose the more recent [or higher?] frequency."

That way you don't need to talk about 30 days, which in my opinion is really a bad constraint, because anything can happen in only one month. I mean, it's summer. What if the respondent just took a 3 weeks vacation and contributed either much more or much less that usual? I don't like the idea of assuming that the respondent will identify the question as inadequate and answer as if it was different.

If we were nitpicking, a not-so-regular contributor could even feel outcast by this question. Maybe they write a full article on their favorite subject once or twice a year and it amounts to nothing according to our question. That's why I think any duration below 6 month or 1 year is not very interesting.

Unrelated thought: There are some pieces of information that can be directly observed from one's log of contributions. For example, whether I participate in discussions about articles. Are there statistics compiled from the raw data and compared with the result of the survey?

-- Rinaku (t · c)

"Pick three"

Just a note that because several people had pointed out (example) that it is not possible for every respondent to "pick three" of the listed problems with Wikipedia culture that they agee with in Q23 (now #18), we have removed that restriction for the new edition, as promised by Ayush here. The resulting inconsistency should be tolerable, as it can be assumed that respondents who now pick less than three would have skipped the question or given a less accurate reply in the former version. Regards, Tbayer (WMF) (talk) 03:22, 17 July 2012 (UTC)

Duration test

The last issue of this survey was completed by over 6000 respondents, meaning that each minute of survey duration corresponds to over 100 hours of Wikipedians' time, and you can help us to use that time judiciously: A test version of the upcoming survey is now up here. Please record the time you needed to complete it, and then tell us below how long it took. Other feedback is still welcome, too (as noted on the page, the questions might see some changes or possible additions). Regards, Tbayer (WMF) (talk) 19:03, 17 July 2012 (UTC)

Three responses from IRC (#wikipedia):

• "took me 6 minutes"
• "07:15.3"
• "my stopwatch says 20 minutes 1 second [...] and 35 hundredths :)"

Regards, Tbayer (WMF) (talk) 19:43, 17 July 2012 (UTC)

Just did the survey - you want the duration here? Took me about 15 minutes, but I knew what I was doing - it could take some people even longer. I think it's length maybe needs to be looked at - some may leave the survey without finishing it because it takes too long - but that's just my observation. Steven Zhang (talk) 23:02, 29 July 2012 (UTC)

"Global South"

Hi,

The term "Global South" is very popular inside the Foundation, but not so popular in the general public. "Developing countries" would be a lot easier to understand. --Amir E. Aharoni (talk) 06:16, 28 July 2012 (UTC)

I thought that Developing countries was an obsolete term - it isn't really applicable to countries where the GDP is below its peak. I prefer Global South. WereSpielChequers (talk) 13:22, 28 July 2012 (UTC)

My understanding is that Global South is the term preferred by the Foundation and used throughout its materials, and is gaining usage in the general public as well. Christine (WMF) (talk) 00:58, 8 August 2012 (UTC)

"Global South" is pretty offensive and it reminds us in Latin America, for instance, of the racist term "sudacas" used by the Spaniards to refer to Latin American immigrants (which BTW, can be even blonde and blue-eyed) when they listen to our accent. And Christine, could you please show us a proof that this term is gaining usage in the general public? To us, it looks like the WMF is imposing a term coined by someone who's not from this side of the world, period. --Jewbask (talk) 14:52, 25 August 2012 (UTC)

I think "developing countries" and "Third World" are slowly going out of fashion, with Global South still much less used, but gaining in popularity. What term would you use instead? --Bence (talk) 15:11, 25 August 2012 (UTC)

"Global South" is a term that started to be used in the Anglican Church in 2003 to describe poorer and more conservative (religiously speaking) nations in the south. I had thought of High-HDI and Low-HDI but it doesn't apply since countries like Argentina have a higher HDI than Russia or Serbia, for instance. OECD and Non-OECD wouldn't apply either, since Chile and Mexico belong to OECD, but not Taiwan or Hong Kong. It's kinda complex..but still, it sounds racist to us. "Developing Countries" sounds more realistic, because in general, our countries are doing that. --Jewbask (talk) 21:11, 27 August 2012 (UTC)

Great to see that the protests of countries from the Global South has still being ignored. Iberocoop chapters have constantly said that GS is a pejorative and unpopular word, still being used in the WMF who knows why. No single international organization not even a country in the so-called GS has used that word, while "developing country/nation" is the most common word. But well, who cares about the opinion of the people from the Global South, right? --B1mbo (talk) 23:05, 2 September 2012 (UTC)

I wasn't involved in the past discussions when this term was chosen, but my understanding is that the term "developing countries" was controversial too, and that "Global South" was regarded as the least worst alternative by many. And while Jewbask's link is interesting, note that it doesn't say that the term was invented by the Anglican Church in 2003, quite the contrary. This is also evident in this version of Bence's link, which shows the sharp rise in usage of "Global South" even more clearly (starting way before 2003). I would much prefer to have a wording that everybody is happy with, but even if the wording hadn't been finalized and translated yet, it seems that a consensus alternative might simply not exist. Regards, Tbayer (WMF) (talk) 23:47, 7 September 2012 (UTC)

I agree with the sentiment above by Carlos and Osmar. I recall some discussion early last year that Global south would be avoided in official communications originating from WMF and eventually, phased out. It might be worth noting that the voices defending the use of the term "Global South", aren't from the "Global South", while the people being referred to here, clearly don't like the term, even going as far as to consider it racist. I suppose it reflects on how term originated and how things are perceived. Interesting. Anyway, I think it is entirely too late at this stage, not that I expect it would have been changed earlier either. Theo10011 (talk) 15:49, 4 September 2012 (UTC)

See Global South. Theo10011 (talk) 09:59, 7 September 2012 (UTC)

While I accept that there are objections to the term (and I didn't author this part of the survey), I find the strong opinions you express a bit surprising, as you seem to have used the term "Global South" quite frequently yourself in the past, without any hint of objection or irony. Was there anything in particular that prompted your change of opinion?

Regards, Tbayer (WMF) (talk) 23:47, 7 September 2012 (UTC)

I'm not sure what seniority you base your high opinions on "I accept there are objections" and so on, you haven't really been around for long at WMF. When people discuss usage of the word in official communications by WMF, they are not referring to you. Try and accept that objections existed far before you even thought of joining the WMF. My objections were never as strong as you have made them at this stage, but they are certainly on record. That objection went to WMF, and not you, which you seem to assume at every turn. So, while WMF found the term, used it extensively for the 3 years, even popularized and communicated the term at every turn, and continues to do at this point, your justification would be that "you used it too"? after 3 years of constant and repeated usage. Anyway, look above maybe more people used it including Carlos, and Osmar, here or in the past, that should be enough to justify your inept reasoning going forward, point that out to them as well. Theo10011 (talk) 12:12, 8 September 2012 (UTC)

It shouldn't be assumed that there was a change of opinion because the term has been used in the past. I believe that's a wrong conclusion. In real life we don't formaly state all the time what we don't agree with, until we do, and since the issue has been raised it should be adressed.--3BRBS (talk) 18:20, 8 September 2012 (UTC)

To be clear, Iberocoop members asked in 2011 to stop using the word "Global South" for being patronizing, outdated, erroneous and pejorative. You can see here when we kindly asked for stop; also, you can see our presentation online (see 1:15:30). WMF is not responsible for creating the term but it has been consistently used even though Latin American, African and Asian members have asked for its removal. "Developing nations" is the most common term: it is an evolving term and denotes how countries are moving (some LatAm countries even have a better GDP than some Europeans) and not are permanently poor just because they are to the south of certain border. We have asked kindly every time with no answer... we are kindly asking it again. --B1mbo (talk) 19:25, 11 September 2012 (UTC)

Another example of the concern presented here: [1]--3BRBS (talk) 14:28, 12 September 2012 (UTC)

Thanks for the links. I had a look at the slide last month, watched the part of the video where you (B1mbo) talked about the issue (btw, here's a direct link to that part: [2]) and read your comments on that talk page. It made me understand the argument of dynamical ("developing") vs. static (fixed geographic delineation) and, in case that wasn't clear, I never had a strong preference about either term. The racist connotations were news to me and I don't see them explained in the links either; I suppose one will have to look elsewhere for more background on that.

I didn't mean to imply that Iberocoop members had uncritically used the term themselves in the past (that remark was directed at somebody else); sorry if you got that impression. However, I couldn't help noting that even Jewbask/Maor X has recently used the term "Global South" himself on behalf of an Iberocoop chapter, sure enough distancing himself from the term by a "so-called". This does of course not make his concerns about racism invalid, but it does point to an argument that Theo has expressed elsewhere: For better or worse, "Global South" is currently the most established term regarding the WMF(-funded) activities in these geographic regions, and when trying to make oneself understood when talking about them (like Theo to mailing list readers, or this survey questionnaire to logged-in Wikimedians) one often feels oneself pulled towards using this term simply because it serves the purpose of being understood best in that context.

Regards, Tbayer (WMF) (talk) 16:21, 25 October 2012 (UTC)

That's one way of distilling my reasoning to its most antithetical, illogical conclusion. I assume I'm the one being referred to as "somebody else", either way, you point out it was for "someone else", apologize, and then go on to point to its usage from Carlos within the next sentence in the same indirect, ineffectual manner. I don't know if all that indirectness is intentional or ordered. You do like to point to old references, and instances of usage, but no one really disputes them to begin with. If you want, I can start using "Non-Tilman world" or "The world previously known as the Global South" on mailing lists, or enter long drawn explanations for your approval, but try and understand that the world doesn't revolve around you - You are perhaps the only one resisting this so strongly, most others agree to vacate the term instantly if it offends. You argument now become "I now know it offends, but I don't care, and that's how I'll refer to 'them' - 'those people over there' - who don't look like me or come from the same continent." your continuous refusal to vacate the term just reasserts this view.

I know it might be hard, but try and understand this minor distinction- You are not the one being referred to! You are offending someone and they are asking you to stop. Your opinion about what offends "them" doesn't matter, it offends. The rest shouldn't matter. ( BTW this debate has been going on long enough, and I'm waiting for someone to breach Godwin's law.)

Let's try this analogy - A racial epithet - Ni***rs for example, used for African-americans in the US. Let's apply your two reasoning-

• a) "You used it too" - In the past, and you might even find some of 'them' still using the term now - That should qualify you?
• b) "There isn't a better term now" (after we established it) - It was introduced by the privileged a while ago, to lump 'them' together - different nationalities, cultures, for all those people from over there who are not like us, and now we just can't go stop!

If you still have a problem understanding, I suggest you try an experiment of using that racial epithet around SF, maybe not the tenderloin district, perhaps in the office (there are enough offensive words for almost all races), and offering the same drawn-out excuses. Who cares if they get offended, your reasoning should be enough. Theo10011 (talk) 11:16, 31 October 2012 (UTC)

India, Brazil, Arabic language

I'd separate these three into their own questions. Many people have opinions (sometimes strong opinions) about one but no opinion about the other. If we want the answer to be a meaningful indication of anything, those should be separate. Asaf Bartov (WMF Grants) talk 22:03, 28 July 2012 (UTC)

We thought about this suggestion, but in the end decided against it - e.g. because of the general desire to limit length (and also, to give somewhat equal space for questions about the various areas of work of the Foundation). Regards, Tbayer (WMF) (talk) 21:19, 8 August 2012 (UTC)

Comments

• The introduction is very wall-of-text. Could it be split over multiple pages?
• "Below is a similar list of activities. Please indicate how often you have participated in the following activities in the last 30 days." - I don't know what this means :). Like, it goes from "series of sorta general options" to "list of even broader ones". Are they meant to change depending on your answers to Q4a?
• For Q17; is "none" an option? Because these are all very, very polarised. It's hard to define an entire community by such polarised terms - if the people I encounter are 49 percent snark and 51 percent yay, does that mean I describe the entire group as "friendly" or "helpful"? Ironholds (talk) 23:00, 29 July 2012 (UTC)

I just tried to shorten the introduction somewhat, but there isn't really much redundancy (and much of it is required legal boilerplate anyway), and added a subsection to make that wall a bit more readable (will still need to check how it looks in Qualtrics). I want to avoid splitting it up because each additional click before the action begins is likely to lose respondents. Regards, Tbayer (WMF) (talk) 21:41, 7 August 2012 (UTC)

My understanding is that the options don't depend on answers to Q4a. As to Q17, you do have to pick two options... if the community gives you snark and yay, you can pick "helpful" and "rude." Or "Friendly" and "Unfriendly." Choosing one does not mean you are not allowed to choose the polar opposite, nor does it invalidate your answer, believe it or not. And, you can always skip the question if you don't feel like any of the answers work for you. Even skipping the question gives us interesting data! Christine (WMF) (talk) 01:51, 8 August 2012 (UTC)

Technology and Networking

Currently this section is very short, and the answers won't really give much useful information. It might be worth expanding it to ask what devices people use to edit, and what they would like to use to edit (focusing on whether they would like to edit using a mobile phone, perhaps?). Mike Peel (talk) 19:10, 31 July 2012 (UTC)

We actually asked those questions (which devices people use to edit and which to read, although not which they would like to use to edit) in the November 2011 survey. However, we took them out this time because we thought that asking them again probably wouldn't yield useful trend analysis and we were looking to cut down on the length of the survey in general. User:Beckien (talk) 19:38, 31 July 2012 (UTC)

See questions 8c and 8d in the technology section https://meta.wikimedia.org/w/index.php?title=Research:Wikipedia_Editors_Survey_November_2011/en&oldid=3065552 User:Beckien (talk) 19:50, 31 July 2012 (UTC)

Ah, makes sense, thanks for pointing that out. Mike Peel (talk) 22:28, 31 July 2012 (UTC)

Spending money, not just the WMF

Question 23 currently reads "We are interested in your opinion on how the Wikimedia Foundation should spend money. If you donated 100 dollars to the Foundation, how would you like the Foundation to allocate money for the following? " However, it's not just the Foundation that receives donations and spends money - it's also chapters. I'd suggest rephrasing to read "We are interested in your opinion on how Wikimedia should spend money. If you donated 100 dollars to Wikimedia, how would you like that money to be allocated for the following?" Thanks. Mike Peel (talk) 19:13, 31 July 2012 (UTC)

(Note that question 23 has since been renamed to P3, Mike was commenting about the questionnaire while it was still in its draft state. As with his question in the following section, I already briefly talked about this comment with Mike in person shortly after he made it, but should have replied here on-wiki with more detail earlier.)

I agree that this might be an useful question - your observation parallels the changed scope of the recently published 2012-13 WMF Annual Plan ("This year for the first time the annual plan incorporates all revenue and spending for the Wikimedia movement (with the exception of “other chapter revenues”)"). But I don't think is wrong to ask this question separately about the Foundation, or, for that matter, about any other organization in the movement, say Wikimedia UK.

And, as already remarked below, this was one of the questions that was already present in the same form in the last survey, and one main motivation for including it was to be able to identify changes/trends. We would have lost that when modifying it. I would also like to point out that the present form of the question was the result of an extensive discussion involving chapter representatives (also in an Internal-l thread last October), as you may recall because you briefly participated yourself, although back then without making the present suggestion.

Regards, Tbayer (WMF) (talk) 12:07, 24 August 2012 (UTC)

In that section, I also see a category "advocacy for free knowledge" or something along those lines that is a bit less terse and focused than "defending editors" or "working on legal cases" missing. It would be interesting to see how much of advocacy the editors would like us to do to defend their work on a broader level. notafish }<';> 10:53, 21 August 2012 (UTC)

I assume you are referring to the choices for P1. Christine may be able to comment further on this, but let me remark that these mostly reflect actual or already planned focus areas of the work of the Foundation - i.e. editors are asked here how it should prioritize among them -, rather than potential new ones, which would mean asking editors how the Foundation should expand these focus areas. Also, as Sue remarked in 2011 (link above) on Effeietsanders' idea to include for opinions about hiring paid editorial staff ("not because I want it to happen, but because I'm curious what our donors think"), there is a risk that adding an option about an activity might be misunderstood, by the press and the community, as an actual intention to take up that activity. And political advocacy, even when limited to the area of free knowledge, is not an entirely uncontentious activity. That being said, I agree that it would be interesting to know, but there is always "P2. Is there anything you think the Foundation should work on that you didn’t see listed above?"

Regards, Tbayer (WMF) (talk) 12:07, 24 August 2012 (UTC)

Hi Tilman. We talked about this, but I'm afraid that I'm still struggling to understand the approach that's being taken here - particularly given my original comment. The WMF keeps making the point that the funds donated to Wikimedia are donations to the Wikimedia movement rather than any single organisation, yet the WMF still ends up asking the question about how it should spend money rather than how the Wikimedia movement should spend money. That makes no sense to me... Thanks. Mike Peel (talk) 21:54, 27 August 2012 (UTC)

Following from this, I've directly modified the phrasing of the question so that it is talking about the Wikimedia movement rather than the WMF specifically. Thanks. Mike Peel (talk) 19:13, 31 August 2012 (UTC)

Mike, the purpose of this question is not to tell the respondent what paradigm the WMF Board of Trustees adopted in its March 2012 FDC resolution for thinking about movement money. I agree with you that it is a good paradigm, but that is beside the point here.

Instead, the purpose is to gauge the respondent's opinions about how the expenses of the Foundation should be distributed, which is a perfectly reasonable question in any scenario, even if one also considers the expenses of other organizations in the movement.

To have the respondent imagine that they themselves donate $100 to the WMF is just a survey technique to make this question less abstract. Yes, if the respondent were to make a donation following a banner appeal on a WMF site during the fundraiser, the money would be considered "movement money" per the Board's resolution, but if one insists on understanding the question that literally, it can also be pointed out that it's still possible to donate directly to WMF or other organizations in the movement, including Wikimedia UK I believe. In any case the question is, and has to be, highly simplified in several other ways. Summarily, I fail to see why this particular simplification should be problematic, even if I would formulate the question differently if it would be posed for the first time in this instance of the survey.

Which brings me to the main point: In your remarks I don't see any consideration of the main reason of repeating this question in the form that it was asked in the last survey (before the Board's resolution): To track changes in Wikimedians' opinions about this topic since the last time. In other words, changing the question removes the main reason it is being asked in the first place. Or to look at it differently: If we assume that it took a responent 1.5 minutes on average to answer this question and distribute "their" $100 among the options, the data from the last survey about this question represents over 150 hours of Wikipedians' time. To discard all this work for the present survey just because of some some rather abstract political considerations seems a bad choice of priorities to me.

Finally, I find it extremely contradictory to on the one hand argue above that respondents must be asked about how chapters should spend money, too (in aggregation with WMF's spending), but on the other hand arguing below that the same respondents are not capable of judging the work that chapters do using that money.

Regards, Tbayer (WMF) (talk) 07:39, 1 September 2012 (UTC)

Chapters

There's a massive section of questions about the WMF, but hardly anything about the chapters, and what there is seems to be put under the WMF. Please could a lot of the questions that are being asked about the WMF also be asked about the local chapter, as well as asking whether they know what a chapter does, and whether they're aware of the chapter in their country (particularly when they've specified that they're in a country where a chapter exists)? Thanks. Mike Peel (talk) 19:20, 31 July 2012 (UTC)

User:Tbayer (WMF) will probably have more information on this (and the above question, this applies to both); I was originally working on a specific survey to detail community satisfaction with the Foundation as a whole, which was merged in with the larger Editor Survey. I do know there are more plans for future surveys, and perhaps a survey dealing specifically with Chapters can be considered. Christine (WMF) (talk) 03:47, 2 August 2012 (UTC)

Is there any update on this? --Bence (talk) 17:31, 20 August 2012 (UTC)

Wondering also why this is a full stage during this survey about the WMF and nothing about the chapters. --Itzike (talk) 17:42, 20 August 2012 (UTC)

Well, the update seems to be that Mike's comment was overlooked, since the survey is supposed to launch in a few days (see Tilman's statement below) and nothing was done to change/correct this. In any case, I find the merging between satisfaction about the WMF and a larger editor survey quite infortunate, I would have found much more interesting and relevant to have satisfaction about the work of Wikimedia organisations ont he one hand and general satisfaction/remarks/feedback about the WM projects on the other. Moreover, I cannot believe that the really out of context question of how the chapters perform is still in this survey this year (under FINAL THOUGHTS). This question, after many questions about what the WMF does, and none whatsoever about what the chapters do is simply bound to ever give the same answer. Can we please take it out once and for all, or put it in a survey where it actually makes sense and is put into context (ie. people are asked if they live in a country where there is a chapter, people are asked whether or not they know what the chapter does etc. the same way this is done for the Foundation?) Thanks. notafish }<';> 08:43, 21 August 2012 (UTC)

The question about "performance", as I noted multiple times, is completely meaningless as far as I can see, and can probably be completely disregarded. As for chapters, given how horrible the questions about chapter were in the past, I think it's better that the WMF (probably to avoid more criticism) skipped them. --Nemo 12:24, 21 August 2012 (UTC)

Comments from Wikimedia Germany

Thank you very much for your outstanding work on this year’s survey! We apologise for not responding earlier, and we promise to engage in the process more timely in the future.

It is great to see that you have further improved the survey and implemented most of last year’s feedback. However, we anticipate severe problems concerning some parts of the survey. Some relate to the specifics of chapter work in general, others are specific to the German translation.

These are the most important points directly affecting our chapter work:

• In section 5 about the Wikimedia Foundation it is unclear to the respondents, whether the work of the chapters is included in (the familiarity F1, support measures F2/F3 and – if intended – general direction F4 of) the Wikimedia Foundation or not. If an inclusion was intended, this should be stated clearly. If not, this should be stated clearly as well. The responses will otherwise reflect a mix of both interpretations.
• In section 5, Priorities P3, the term “community work” seems to be ambiguous in the English version. What exactly is meant by this term? Because of this ambiguity, the German translation “Gemeinschaftsarbeit” is definitely misleading. If this question is actually going to be used to allocate funds in one way or the other, this will lead to severe bias.
• In section 5, Final Thoughts T3, the question concerning the chapters cannot be properly answered by respondents that live in a country without any chapters or even by respondents that do not know that chapters exist or what their purpose is. Again, this will lead to severe bias rendering the responses meaningless. (This question falsely will generate answers that look like actual opinions about the work of chapters, where in fact, the respondents only moments before might not even have know what chapters do or that they even exist.) We second Delphine’s suggestion to either completely strike this question off the survey or add a filter question where respondents are asked whether they live in a country with a Wikimedia chapter (Yes/No/Don’t know). This question should introduce the term “chapter” and give examples that are relevant to the respondent’s country of residence.

We find it very important that all language versions can build upon the tremendous work that you have done with the survey. It would be a real shame if the responses to several questions could not be used for methodical reasons the chapter questions) or simply because they cannot be understood in German.

We have offered Tilman the help of our research and translation staff to give more detailled feedback.--Nicole Ebber (WMDE) (talk) 16:30, 21 August 2012 (UTC)

Thanks for the feedback, Nicole, and I'd likewise apologize for not responding here earlier.

First, let me confirm in public that I did indeed receive this friendly offer by email. Apparently there had been some misunderstandings about the German translation that the WMDE team had been reading here on Meta, in particular that it was

1. a draft
2. in the process of being reviewed by volunteer translators (see also #Translation status and NOTICE ABOUT TRANSLATIONS)
3. freely editable (as are most pages on Meta).

But those were cleared up subsequently, and I'm grateful to the WMDE staff members who have since participated in the translation review process to which I had invited on the day before your comment, as I am to the at least six volunteers from the German-speaking community who have done the same. I hope you were OK with the more than 100 further changes that these community members made to your version. (I'd also like to note that as a native German speaker and longtime community member myself, I would have felt quite comfortable in checking for questions that "cannot be understood in German", but considering the many other tasks involved in this survey, everyone's support was much appreciated.)

• "In section 5, Priorities P3, the term “community work” seems to be ambiguous in the English version. What exactly is meant by this term? Because of this ambiguity, the German translation “Gemeinschaftsarbeit” is definitely misleading."

The translation that you criticized as "misleading" was chosen for the December 2011 survey by an experienced German Wikipedian; we can assume she is at least as familiar with the language used in the German community as you and me. That being said, perhaps her translation wasn't optimal, thanks for the efforts to improve it. I agree that it's not always good to substitute German terms for English ones (even though it has to be said that this volunteer was not the only one to do so for "community" in this questionnaire).

You did not explain why you considered the term "community work" to be ambiguous in English. But I surmise that this was because it can also mean volunteer work by employees of an organization for charitable purposes (other than that of their own employer). While many WMF employees are doing such volunteer work outside of their working hours, I think that the financial context of the question makes it reasonable clear that the intended meaning was quite different, based on the interpretation of "community" as "community of editors on a Wikimedia projects" which can be assumed to be common among the target group of the survey, i.e. members of this exact same community.

• If this question is actually going to be used to allocate funds in one way or the other, this will lead to severe bias.

You do not seem to have been aware that this question had been taken verbatim from the December 2011 survey. Back then, it had been reworded from its April 2011 version following an extensive discussion involving many chapter representatives, after Sue had solicited feedback on its wording (in an Internal-l thread comprising 37 postings, and here on Meta). Despite this lively discussion touching various aspects of this question, none of the many comments by chapter representatives objected to the term "community work". What's more, the current wording that you are concerned about was actually authored by a co-founder and former vice chairman and Executive Director of Wikimedia Germany, which makes me even more interested in the factual basis for your claim that it could endanger Wikimedia Germany's work.

Also, your statement did not give the impression that it was written by someone who is very familiar with the statistical meaning of the word "bias". Was it assuming that there can only be bias in a survey if funding decisions are subsequently based on it? And where would that "severe bias" lean toward, and why?

In any case, as Sue has since tried to clarify in a mailing list posting, the present survey is not run for the purpose of using its results to determine decisions regarding funding - this is not a budget vote. This is perhaps in contrast to other surveys and research reports - such as WMDE's own Spendwerk report - that were produced on behalf of various movement organizations during the fundraising debates that ended with the Board's decision earlier this year (during which period Sue's above mentioned feedback request had happened).

In other words, the concerns about possible financial losses for Wikimedia Deutschland, which appear to have contributed to its decision to allocate staff time for participating in this discussion, would appear unfounded. Apologies if this had been unclear in the beginning, and led to unnecessary expenses for WMDE. (Perhaps the misunderstanding about the financial implications had been furthered by the Board of Trustees' commitment to integrate community feedback into the FDC funding process, but that was achieved differently. The present question had never been designed for that purpose, nor would it have been suitable for it.)

• "In section 5 about the Wikimedia Foundation it is unclear to the respondents, whether the work of the chapters is included in (the familiarity F1, support measures F2/F3 and – if intended – general direction F4 of) the Wikimedia Foundation or not."

Wikimedia chapters are organizations in their own right which are separate and independent from the Wikimedia Foundation. The target group of this survey are active editors, and several questions assume some basic wiki- or Wikimedia-specific knowledge. To cite some other examples that already occurred in the previous surveys: Respondents need to know what a featured article is, what reverting means, or indeed that Wikipedia is freely editable - among the general populations, not even the last one woudl be a fait accompli. With each of these, it is expected - like in basically every survey - that there may be a small percentage of respondents which are unaware of that fact, but that this does not affect the quality of results too much, or that it can be controlled for - indeed question F1 would largely seem to allow to do that. You appear to be assuming that there are many active editors who on the one hand know about the chapters and their work - otherwise this confusion wouldn't even arise -, but on the other hand confound them with another organization which has a different name, even after a short description of that organization has been provided. I am not aware of an empirical basis for this concern, nor have I seen it brought up among the many comments about the previous two surveys, which already included questions both regarding the Foundation and the chapters.

Also, for each of the above three points (responding to the third separately, as it was the only one brought up by others as well), you did not describe any mechanism by which it could be "directly affecting [y]our chapter work". Summarily, I am proceeding under the assumption that these won't be major issues. But if it indeed turns out that Wikimedia Germany's great work suffers measurable damage as a result of this survey and/or the two previous ones, then by all means let us know what happened.

• Finally, while I of course fully acknowledge that thoughtful evaluations and responses take time, I have to say that I found it puzzling that after the German chapter reported on this upcoming survey on its official blog ("Feedback about the questions is welcome. Changes are still possible for a short time; translations are planned to start this week"), a full 19 days including the public review period passed before it first mentioned the "serious problems" it saw, despite fearing that its own activities were threatened by them.

Regards, Tbayer (WMF) (talk) 23:30, 30 October 2012 (UTC)

I totally agree with Nicole and Delphine, this section talks too much about the WMF (and believe me, a lot of people doesn't even know the WMF exists) and nothing about the chapters, which are the ones performing activities at a local level. And perhaps can be read as "not really". --Jewbask (talk) 14:55, 25 August 2012 (UTC)

Mike, I fail to understand why the onus is on WMF to inform the editors about chapters? The survey is titled the Editor survey, rightfully so, it is questioning editors about the project and through that relation, WMF. Chapters aren't engaged in hosting or running the projects. Majority of the chapters are not directly linked to editors, speaking just for anglophones, WMUK can not hold any authority to be addressed on behalf of English Wikipedians. Theo10011 (talk) 16:32, 25 August 2012 (UTC)

Please don't misrepresent my views. I expect the onus to be on the organiser of any Wikimedia survey to make sure that it is as unbiased as possible. In this case, chapters are very much involved in community and editor relations, and hence they should be fairly represented in any survey that is asked to the community. I have no clue what you mean, or are intending to imply, by your statement about WMUK and authority. Thanks. Mike Peel (talk) 21:30, 27 August 2012 (UTC)

I don't think I was misrepresenting your views, intentionally anyway. I was implying that Chapters won't be the only entities anymore representing WMF, why stop at just chapters. The other point was about English Wikipedia, and the logistics of targeting only UK based editors which might not represent a large subset of the larger editor base, only to inform or question them about a local chapter. Anyway, I wasn't sure what I was thinking at the time myself, Please feel free to disregard it. Theo10011 (talk) 19:37, 28 August 2012 (UTC)

Mike, it seems this is conflating "unbiased" in the sense in which is usually used in the context of surveys (e.g. not favoring certain answers to questions) with evenness of coverage of certain topics (i.e. which areas to ask questions about). And even if the questionnaire spent an equal amount of bytes on Foundation and chapters, it would still leave out many other important topic.

Also regarding "should be fairly represented", let us remember that most surveys of Wikipedia editors are not done by WMF or chapters, but by independent researchers (e.g. I just reviewed one such study in the new issue of the Wikimedia Research Newsletter, where the researchers had to resort to a somewhat awkward method of contacting respondents). The Foundation does not make demands from the researchers conducting such surveys that WMF "should be fairly represented" in their questionnaires; it would be a very strange thing for us to do.

Regards, Tbayer (WMF) (talk) 14:11, 30 August 2012 (UTC)

Proposal for an independent Editor Survey that addresses local impact and Chapter engagement

I had trouble finding any mention of the chapters in this list. Sadly, this Editor Survey will have no tangible value in understanding the impact of the Chapters or the work we do. If the Chapters would like a survey that meaningfully addresses local impact, then it appears (based on a lack of response or interest here from the Foundation over the last month), that the Chapters may have to commission our own independently of the Foundation. Fæ (talk) 15:51, 25 August 2012 (UTC)

Why don't you go ahead. Theo10011 (talk) 16:32, 25 August 2012 (UTC)

As soon as the FDC gives us a grant to do it. Perhaps you would like to help prepare a proposal Theo? We can base a budget on what this survey costs. --Fæ (talk) 16:37, 25 August 2012 (UTC)

No. Thanks for asking though. I'm done preparing budgets. ;) I just enjoyed the irony of asking WMF for a grant to fund another survey because the one they are running isn't independent enough. Anyway, I'm sure the future FDC, and its 30 odd proposed members, would fund that one right through. Theo10011 (talk) 16:59, 25 August 2012 (UTC)

I already commented on a similar question in the IRC office hour shortly before Mike made his suggestion above, and shortly afterwards briefly talked to Mike about it in person (so much for notafish's and Fæ's speculations). But thanks for reminding me to respond here too, and apologies we didn't do so earlier - the translation process and other necessary preparationg for the survey's launch have been quite time-consuming this week.

To recap what was already said earlier, the idea is that after we get the editor surveys going again with this one, they will be run more frequently from now on (aiming at a quarterly schedule), in a more lightweight version, and with a different focus for each one. We are fully aware that there are a lot of interesting possible questions which are left out of the present survey - not just regarding chapters, but also, for example, our WMF colleagues working on new software features would like to get feedback on them, as noted above; but we had to tell them to wait for an upcoming survey, too. We also plan to reach out to academic researchers and to the editing communities themselves to have them suggest questions in the future. Regarding chapters again, here's what I (HaeB) said in the IRC office hour:

Jul 31 10:17:38 <Nemo_bis> 2) Maybe you want to comment why chapters are no longer mentioned (I know, I suppose, but it's worth noting)

Jul 31 10:18:04 <HaeB> some of those actually arent' in this edition any more (mostly because it was questions where a one-time answer was enough, and folowing trends wasn't too important)

Jul 31 10:19:00 <Nemo_bis> I appreciate the "99 I don’t know" option btw

Jul 31 10:20:03 <HaeB> so that question actually still retains the part about chapters

Jul 31 10:20:37 * Fluff|brb is now known as Fluffernutter

Jul 31 10:21:04 <Nemo_bis> oh, right, there's still that one

Jul 31 10:21:06 <rneumann> question Q10 asks about participation in chapter work

Jul 31 10:21:17 <HaeB> but yes, the emphasis this time is on satisfaction/dissatisfaction with wmf ;)

Jul 31 10:21:23 <rneumann> it asks how often people have done this activity in the past 30 days

Jul 31 10:21:50 <HaeB> again, we are already thinking forward to the next survey where we will have room for other things

Jul 31 10:22:51 <HaeB> one person from an european chapter suggested to ask "Please tell us what you think were the top three/five/few achievements/contributions of [your country's] chapter in the past year. "

Jul 31 10:23:27 <HaeB> i think that generates technical problmes (it's hard to evaluate free-form answers if you have >6000 respondents)

Jul 31 10:23:44 <HaeB> but we may look to include something similar

Summarily, I'm absolutely open to the idea of including a focus on chapters in one of the upcoming surveys, and I will be happy to delve into the discussion about what would be good questions to ask once I have a little more time after the launch of this one. However, I sense a notion of "wherever there is WMF there must be chapters" in some comments, which I find a little strange. For example because, on the other hand, some of the same comments seem to suggest chapters are not a suitable topic for an international survey targeted at all logged-in editors such as this one (I disagree). But also because it does not seem to acknowledge that surveys like this are a huge demand on editors' time and can't be expanded at will - adding a question that takes a respondent one minute to answer on the average corresponds to asking for more 100 additional hours of Wikimedians' time. So it does make sense to restrict the focus of each survey, and separate topics over subsequent surveys.

I would also like to point out, for the benefit of those who have not yet read the Q&A for this survey and the "notice about translations" in the questionnaire, that we are at a point where it is extremely difficult and costly (in terms of both staff and volunteer translator time) to change the questionnaire, as it has been finalized a while ago and the translation review process is wrapping up now. But again, I'm looking forward to including chapters-related questions for the upcoming surveys.

Regards, Tbayer (WMF) (talk) 17:02, 25 August 2012 (UTC)

Thanks for the reply. As for Notafish and my "speculations", I'm afraid we can only go by the facts presented, my noting an absence of follow-up comment after Christine's initial note here over the last month was not speculation, it was a statement of precisely what can be seen. I cannot feel guilty for being unaware of the contents of an IRC discussion back in July, in fact checking my diary I was in a Wikimedia UK telecon at that exact time. I look forward to a being given a firm commitment on a date for a survey that will address chapter engagement. Perhaps you could provide a date for such a survey to run and we might then conclude this discussion thread; as if there is a date, then Wikimedians interested in the chapter view but who may not keep an eye on the Editor Survey can plan to participate in plenty of time. Obviously, if a future survey that does what is expected cannot be delivered until 2014, then it may still be relevant to commission an alternative in 2013. --Fæ (talk) 17:25, 25 August 2012 (UTC)

Hi Fæ, I know you were really busy at the time of the IRC office hour due to important changes going on at Wikimedia UK, and of course you were not required to attend. However, the log of the IRC office hour was published at IRC office hours/Office hours 2012-07-31 on July 31 and has also been linked from the survey page.

I think that Wikimedia-l was a good place to announce the survey last month, and for Wikimedians who want to make sure they do not miss any new developments of movement-wide relevance, I would recommend subscribing to that list. But for future surveys, I'm open to suggestions for additional venue for the announcement.

The decision about the future surveys is not mine alone to make, but I can confirm that the current thinking, set out by Barry, is to run them quarterly (of course this may need some adjustment in some cases, e.g. due to the fundraiser, but postponing until 2014 is definitely not on the table), and that I had a conversation with Barry and Anasuya in July - not long after Wikimania - where we agreed that including questions specifically about chapters would be a good idea for the next survey. (As many may know, along with the rest of the WMF Communications team I am changing over from Barry's department over to the LCA department, which already provided the part of the current survey that is being discussed here, but I do not expect this to change the equation.)

Regards, Tbayer (WMF) (talk) 21:47, 25 August 2012 (UTC)

(Did anyone else immediately think of Hitchhiker's Guide to the Galaxy at this point? :P ) Orderinchaos (talk) 14:10, 26 August 2012 (UTC)

lol, I did. ;) Theo10011 (talk) 13:21, 27 August 2012 (UTC)

Hi Orderinchaos, this was an honest attempt to give Fae (and other readers) as much relevant information about the Foundation's internal decision process as possible. I hope your reaction does not come out of a dismissive attitude towards institutional transparency in general. Regards, Tbayer (WMF) (talk) 14:11, 30 August 2012 (UTC)

Mine does. Or it might be to your ineptitude at communication or to the joke of institutional transparency being advertised above which might suit vogons more.Theo10011 (talk) 12:18, 8 September 2012 (UTC)

Tilman, "in terms of both staff and volunteer translator time", from what I know, the foundation paid for translation and didn't uses the movement translator network for this survey, from some unknown reason to paid extra money. --Itzike (talk) 19:12, 25 August 2012 (UTC)

Itzik, at this point I would much appreciate it if you could spend a few minutes to familiarize yourself with the survey's process. For example, just reading the "notice about translations" in the questionnaire that you have been commenting about would have saved yourself the embarrassment of making the false statement "didn't uses the movement translator network for this survey" in public. Yes, we used a inexpensive and fast external translation service to start off the process and make sure that the most important languages were covered (the last editor surveys had likewise commissioned some paid translations toward that goal). But their subsequent review and editing by volunteer translators - who are likely to be more familiar with movement-specific terms than the professional ones - was an integral part of the process from the beginning; as was the opportunity to provide volunteer translations in additional languages. Involving the movement translator network was the whole point of providing the questionnaire translation here on Meta; and I wrote a new bot specifically for that. One could also just have read my August 20 comment further below on this page, or looked at the version history of one of the translation pages (for example the Hebrew translation which was entirely done by volunteers), or checked the edits of my account from earlier this week to see hundreds of translation notifications sent out to volunteers (example). Regards, Tbayer (WMF) (talk) 21:47, 25 August 2012 (UTC)

I have been noticing more and more unnecessary snark in your comments. I don't know how you are first calling his statement false, then proceeding to prove him right in the following sentence. By your admission above, an external service was used, his claim wasn't disproved in that alone, you just redirected, that the external translations thus provided were up for review on Meta, while some less "important languages" were translated by volunteers. It would have been a rather simple explanation without the use of polemics and more tact.

Well, Theo...didn't you expect it? --Jewbask (talk) 03:47, 29 August 2012 (UTC)

Your IRC log and explanation above, seems to imply that every one on one conversation or comment, or your contribution history should be followed for answers, rather than direct questions that were posted on the designated page, like this one where the question mike raised remained unanswered while others were answered. Theo10011 (talk) 19:29, 28 August 2012 (UTC)

Theo, as it is clearly stated above, I wasn't referring to the use of external services, but to the claim that WMF "didn't uses the movement translator network for this survey", which I hope you are not going to defend. Maybe you are right that the response could have been worded a bit friendlier, but it was not the first time that someone was spreading inaccurate claims about the survey, and this one was a bit exasperating to read right after hundreds of notifications had been sent out to that network and its volunteers had done work on translations of the survey in over 10 languages - especially because it would have been really easy to find out that the opposite was true. Not necessarily by following my contribution history, as you are pointing out correctly, but surely by just reading the prominent "NOTICE ABOUT TRANSLATIONS" (sic) in the very questionnaire they were commenting on, or my comment titled Translation status right here on what you call the "designated page" (where apparently the commenter was eagerly monitoring my contributions for a response to this section). Having the courtesy to spend these one or two minutes to inform oneself about the matter which one is commenting on helps to convince others about the seriousness of the concerns expressed. Regards, Tbayer (WMF) (talk) 14:11, 30 August 2012 (UTC)

I've removed the question about chapter performance, given how it has been repeatedly pointed out (both here and on the mailing lists) how flawed this question is. It can be added back if/when we have a survey that fairly covers chapter activities. Thanks. Mike Peel (talk) 19:15, 31 August 2012 (UTC)

"performance of the Wikimedia chapters, in contributing to the Wikimedia movement" question

To recap: Like the previous two surveys in April 2011 and December 2011, the present survey asks editors to rate the contributions of four different parts of the Wikimedia movement: Volunteers overall, the Foundation, the chapters, and the respondent himself/herself. It seems that in 2011 when this question was first published, no concerns regarding the inclusion of chapters in that list prior had been voiced originally, before the publication of the results of the first survey. But it's true that since then, many chapter members have objected to this. During the drafting of the questionnaire for the present survey, I tried to examine these arguments in detail (including reaching out to one of their proponents in person at Wikimania this summer). While there will likely remain disagreement, I promised earlier to outline the rationale, as I see it, for including this item in this survey as well, responding fuller those objections before the survey launches. Many of them were voiced outside of this page, e.g. for some reason in a more recent thread on the private mailing list Internal-l, but I don't see a reason not to continue this discussion here in public. (As noted below, we did make a change to address one part of this feedback, resolving concerns about possible bias due to the placement of the question which had been expressed by Delphine above, among others.)

• Should editors from countries or regions where there is no chapter be allowed to rate the work of chapters in general?: This has been by far the most frequently made argument against the inclusion of chapters in this question. I'll try to respond to it in the example of the WMDE staff comment above: "In section 5, Final Thoughts T3, the question concerning the chapters cannot be properly answered by respondents that live in a country without any chapters or even by respondents that do not know that chapters exist or what their purpose is."
  • First, it appears that like a lot of the criticism, the cited comment overlooked the fact that an explicit "I don’t know" answer option had already been added to the question. (Almost all questions in this survey can be skipped without answering, but only few include such an explicit "don't know" or "Prefer not to say" option; the question about gender being another example.) In other words, we are talking about excluding the opinion of editors who themselves feel qualified enough to answer the question, and we would expect that honest "respondents that do not know that chapters exist or what their purpose is" will recluse themselves from judging their performance. This renders the second part of the above quoted objection baseless. Admittedly, dishonest editors will still be able to lie, but that risk exists with any survey question.
  • Second, the argument that editors who live in a country/region without a chapter (e.g. myself) cannot have a valid opinion about the performance of chapters, and must therefore be excluded from this question, ignores the global importance of chapters' work. I will concede that I understand part of the logic - let me summarize: The research and translation staff of Wikimedia Germany appears to hold that a chapter's work has no global impact or importance beyond its own country, because its activities are localized there. Therefore, it is assumed that while the global performance rating makes sense for the Foundation, because it is relevant to every Wikimedian, it should not be asked for chapters, because each of them is irrelevant to all Wikimedians except those from one country. - I agree that having an account on Wikimedia projects (the defining characteristic of the target audience of this survey) entails prima facie a much more direct relation with the Foundation than with any chapter; e.g. simply because it runs the servers that they are working on. However, I disagree in that I think that a chapter's work is relevant beyond the corresponding nation. For once, simply because the Wikimedia projects, support of which is, IIRC, a main purpose of chapters, are always international. This is especially evident at the Wikimania conferences, which have so far been organized or supported by at least five different chapters. What's more, some chapter activities are even physically extending far beyond the border of the chapter country, e.g. the admirable projects by Wikimedia France or Wikimedia Israel in Africa. Finally, if I'm informed correctly, some of the work of Wikimedia Germany itself is explicitly intended to support the global editing community: To put it bluntly, I am hopeful that Nicole's argument is not taken seriously even within her own organization, or else the Wikidata developers and Toolserver admins would be need to start dismissing feedback about their work by all non-German Wikimedians right now ;)
  • Finally, the general notion that respondents must not be asked about their opinion about actors in faraway countries which they may or may not be informed about does not appear to be rooted in the actual practices of professional surveys. See e.g. this recent survey from Pew Research (which accepted "Don't know" levels of up to 73%, when Indian respondents were asked about Hu Jintao).
• Used to disparage chapters?: Some have voiced concern that the question's results in the last two surveys have been used to disparage chapters or put them in a negative light in public. While I disagree that this was the case in the cited example (Sue's Wikimania keynote), it's true that for practically any such survey, results can be cited selectively, and that those from opinion surveys in particular can be used to criticize people, organizations or concepts. But distorted representations or insensitive comments by others can't be blamed on the survey itself. And actually, in the present survey, this risk extends much more to the Foundation itself and its various departments and teams than to chapters: With this question block, we are putting ourselves out there, and will likely receive some judgements and opinions from respondent that we disagree with, or think are misinformed. In any case, I do understand the frustration of chapter members who are doing excellent work, in case they feel that a judgment about a large group is applied to them personally without discretion. I hope the past discussions about this survey may have served to sensitize people about interpreting its results carefully and "with a big grain of salt", as Sue said somewhere.
• "The Wikimedia chapters" can't be judged a whole?: This argument was brought up fairly recently and I sympathize somewhat with it. Again, elsewhere in the same question my own work for the Foundation will be judged in aggregation with that of more than 100 other employees, for good or bad ;) However, professional surveys regularly ask for opinions about organizations or groups that are far larger and more complex that even WMF and chapters together. In fact, "Wikimedia volunteers overall" are one such group and the first part of the performance question asks about their contribution as well, but there has been no concern about this so far. In this point like in some others, I feel that people are pointing out limitations that apply to such surveys in general, but selectively apply them to one particular question.
• Professionalism: Also, I would like to point out that the contested question had been authored by a sociology PhD, who already had several years of professional experience in quantitative research and conducting surveys in various countries when she joined the Foundation staff. This does not mean at all that it can't be criticized with sound arguments, of course. But perhaps it explains scepticism towards arguments made from authority against the validity of this question ("no survey ever does X") without either naming the critic's own professional credentials or citing professional or academic literature about surveys.
• Evaluate results per chapter/non-chapter countries?: E.g. above, Nicole suggested to "add a filter question where respondents are asked whether they live in a country with a Wikimedia chapter (Yes/No/Don’t know)". Again, I'm not sure she was aware that a preceding question already asks where the respondent lives. We can indeed use this for a more fine-grained analysis of the chapters rating results, I'm going to outline something below (this doesn't need to be decided until after the survey has run). This will also help to explore the concern that admitting all countries introduced a negative bias against chapters. I haven't seen a pertinent explanation for a mechanism that would introduce such a bias. (Someone already did such an analysis using the public datasets from the previous two surveys; to me the preliminary results did not appear to yield strong evidence for such a bias. I hope they will be published somewhere.)

There is more aspects to this, and I'm happy to discuss them as well as the survey progresses. And I think there is time for that; I don't expect we will include the same question in the next survey when the plan progresses to run these more frequent from now on.

Regards, Tbayer (WMF) (talk) 23:30, 30 October 2012 (UTC)

Bugs

A few little things :

• In question P1 I saw an empty line at the top of the table.
• In question 17b I find the progression of the possible answers weird: there is a huge gap between "Very Happy" and "Neither Happy or Unhappy", and between the latter and "Very Unhappy". Instead of I think it should be more like (Very Happy / Happy / Neither Happy or Unhappy / Unhappy / Very Unhappy).
• It's sad that one can't go back to a previous page in the survey.

--Rinaku (t · c) 12:04, 2 August 2012 (UTC)

Some of these (like the not being able to go back) may be limitations of the survey software, but I'll doublecheck that. Thanks! Christine (WMF) (talk) 19:32, 7 August 2012 (UTC)

Hopefully this helps answer your questions:

• The top of table in P1 is supposed to blank in this question's current design
• I'm not sure what the original reasoning was for this particular wording, but we've kept it in this survey because it was used in past surveys and we wanted in order to preserve as much continuity as possible
• In the November survey I believe it was decided to not include a back button enabling respondents to change their answers to previous questions or previous sections, but we can add one this time if the consensus is that there should be one

Thanks for the feedback! Beckien (talk) 20:02, 7 August 2012 (UTC)

Actually, I take that back. There was one in the November survey, and we'll add one to this survey as well. Beckien (talk) 14:07, 8 August 2012 (UTC)

Allocate $100 (Q23)

"If you donated 100 dollars to the Foundation, how would you like the Foundation to allocate money for the following? (Please ensure that all the responses add up to $100.)"

I think this is not a good question. It is interesting to know how people think the money should be spend, but here I cannot give a good answer because I don't know how much each mission costs. For example, I think the primary goal of the foundation is to keep the website online, but I don't know whether this currently takes 30% or 80% of the Foundation's budget.

I think the question could be replaced by one of 2 better ones:

• Ask to sort the mission from the most important to the least important, without talking about money.
• Ask whether each mission should get more or less money (compared to the others) than it currently gets. (The question could tell how the money is curently allocated.)

--Rinaku (t · c) 12:43, 2 August 2012 (UTC)

This is a question that was also asked in the last incarnation of the survey, I believe, and we kept it this time around (and in its same form) to be able to track change over time. It's okay if you don't know the percentage of the budget each of these items takes; this question is more about understanding what you think the priorities of the Foundation should be. If you think that the primary goal is to keep the sites online and that 80% of the budget should be geared at that, then by all means, say so! Christine (WMF) (talk) 19:31, 7 August 2012 (UTC)

The point is, it's not that keeping the sites online "should" use 80% of the budget. On the one hand, if it only required 20% of the budget, you wouldn't try to pay twice the price just for fun. On the other hand, if this mission required 95% of the budget, you would do it anyway, you wouldn't take the sites down to do something else with the money... at least I hope so :) But do what you can with the question. --Rinaku (t · c) 22:19, 7 August 2012 (UTC)

Features, design, community

For purely selfish reasons, I'd like to see at least a line or two in the WMF technology section about our work experimenting with and designing new features, as well as working with editors to revamp community processes like page patrol, article creation, and user messaging. From what I heard at Wikimania (and what I continue to hear on mailing lists and in onwiki discussions), most editors have no idea that this is something for which we're allocating engineering resources, and I'd be interested to see if that lack of awareness is representative of the broader editing population. Maryana (WMF) (talk) 16:52, 2 August 2012 (UTC)

Due to the time constraints, I'm not certain we can add any other questions. But, I think that the initiatives you're working on would be a good topic for a future survey! Christine (WMF) (talk) 19:23, 7 August 2012 (UTC)

What language is primary, contribute, read

The first set of questions in three parts feels a bit cumbersome. Is it possible for one to select all the languages they speak as the first question, and then the next page can list those selected languages with radio button on whether you read or contribute to Wikipedia in those languages. This may have the added benefit of identifying the untapped reserve of editor talent in various languages.

It is a bit cumbersome, it's mainly this way so we can track changes from the last survey, which was also done this way. It is a pretty big pain point though, and something to look into fixing in the next go 'round of the survey (and future ones). Christine (WMF) (talk) 00:01, 8 August 2012 (UTC)

Agreed. I suppose one potential advantage of doing it this way is that people may still read (although probably not contribute) in languages that they don't call "primary" (respondent might differentiate between primary languages and all languages spoken--for instance, someone might understand a language but not use it in daily life and therefore not label it as "primary"). This way doesn't prevent respondents from choosing non-primary languages in the reading/contributing questions, esp. since past surveys didn't allow respondents to move backwards and change answers. This is, of course, a very small possibility and it creates a rather cumbersome structure, so this might be something to consider in future surveys. Beckien (talk) 20:16, 7 August 2012 (UTC)

I didn't understand the "primary" thing. I thought it means native language. You should use more standerd termes: native, foreign, languages you contribute in Wikimedia. Otherwise, the survery has really good questions. Thank you! --NaBUru38 (talk) 00:53, 31 October 2012 (UTC)

typo in introduction

super small corrections:

1. "You have made Wikipedia what it is and your input will help make our projects even better." : there should be a comma after "is" and "and your..."
2. "We realize that this is a significant request of your time and we want..." : comma between "your time" and "we want."

Jwild (talk) 19:42, 6 August 2012 (UTC)

Super small, are still super helpful! Thanks! Christine (WMF) (talk) 19:23, 7 August 2012 (UTC)

Fixed, thanks! Regards, Tbayer (WMF) (talk) 21:41, 7 August 2012 (UTC)

Translation status

The survey is intended to be launched within the next days. It will be seen by many thousands of Wikimedians, and to help make sure that they are not confronted with confusing or flawed questionnaires, please help reviewing the existing translations and mark them as "Ready", or leave a comment on their status below. Thanks!

About the translations for the survey's questionnaire and CentralNotice banner, see also the notice about translations and the message group statistics.

The questionnaire consists of an old part, where translations can be reused from the last survey if available, and a new part. The externally provided translations for the new part into German, Spanish, French, Russian, Japanese, Italian, Chinese, Portuguese, Polish and Arabic have all been imported (using a new bot written specifically for this purpose), alongside with their counterparts from the previous survey, except in Polish and Portuguese, where those old translations were not directly available on Meta - but they will be imported shortly as well.

Translations in some other languages have already been provided and they are welcome until shortly after the launch of the survey, as long as the language is available in the interface of Qualtrics, the survey platform we are using (see the list). If you would like to provide a translation where the old part is available but has not been imported yet, feel free to start it and ping me so that I run the bot for that language.

Regards, Tbayer (WMF) (talk) 10:04, 20 August 2012 (UTC)

Most translations have been reviewed by now, but as it might have been noticed since last week, the launch of the survey has been delayed for various reasons, including taking more time to respond to feedback here and on Internal-l, a Unicode bug (thanks to Evan for helping to resolve it), and other aspects. At the moment the goal is to launch early next week.

Regards, Tbayer (WMF) (talk) 09:22, 8 September 2012 (UTC)

A missing verb?

Is this sentence correct ? "You can more information about Wikimedia Foundation initiatives here" in the exit screen. Thank you --Cquoi (talk) 17:05, 23 August 2012 (UTC)

Fixed, thanks! Fortunately, it seems it had been silently corrected (or even not been noticed in the first place) by the translators. Regards, Tbayer (WMF) (talk) 17:14, 23 August 2012 (UTC)

I hope the extraneous bits were copy-edited out beyond this version. Tony (talk) 10:48, 29 August 2012 (UTC)

Position of the "how would you rate the performance ..." question

In recent weeks, after the survey questionnaire had been finalized, several people from Wikimedia chapters have voiced concern (mainly on the private Internal-l mailing list; I hope that that discussion can be moved into a public venue) that question where respondents can express their opinion about the performance of "the different participants of the Wikimedia community" (volunteers overall, Foundation, chapters, oneself) is preceded with several other questions that ask in detail about activities by the Foundation.

While I am not quite convinced that this introduces a clear positive or negative bias for the question about the Foundation compared to the other three, in the survey team we have been looking into addressing these concerns without having to restart the translation process, and have decided that it is an acceptable compromise to move this question up to right before section V, which asks those detailed questions about WMF, so that respondents will only see them after having already given the performance rating. (The English version of the questionnaire had already been imported to Qualtrics, and we made the change there first, as people who over the last week or so have tried out the Qualtrics link given above might already have noticed; for transparency and consistency I'm now adjusting the version of the questionnaire here on Meta too; this should not invalidate existing translations, fingers crossed.)

The downside of this change is that it makes the questionnaire a tiny bit less consistent regarding the grouping into sections: The performance question is still being asked of all respondents including Commons users, but section IV only from those who are coming from a Wikipedia site. But that seems a tolerable price to pay for addressing these concerns of possible bias.

Regards, Tbayer (WMF) (talk) 15:01, 9 September 2012 (UTC)

It's nice to see a willing of finding a compromise. But I'm still skeptical about the obstinacy too keep a single controversial question, whereas the simple removal of this question would have cause no problem of translation and no disturbance to the consistence of the survey...--Chandres (talk) 08:20, 10 September 2012 (UTC)

Update on the timeline

Hi, is there any update on the timeline for this survey? Thanks, --Nicole Ebber (WMDE) (talk) 11:54, 4 October 2012 (UTC)

Thanks for asking! After various events and other projects intervened, work on the remaining technical tasks was taken up again and we currently aim to launch tomorrow PT (i.e. before Saturday morning UTC). Before that, I'm also still going to post some outstanding replies to feedback. Regards, Tbayer (WMF) (talk) 16:21, 25 October 2012 (UTC)

Russian version

The Russian version needs few small grammatical corrections. Gamliel Fishkin (talk) 00:48, 31 October 2012 (UTC)

Please correct them here and we will update the live survey version from there. Many thanks! Regards, Tbayer (WMF) (talk) 01:56, 31 October 2012 (UTC)

Less or more compared to half a year ago

Hi, there is a question:

“If you think back to half a year ago: Were you more or less active on Wikipedia compared to now?”

If and only if I select “less” the next question is:

“Why did you become less active on Wikipedia? Please choose all that apply.”

That does not make sense. If I were less active half a year ago, I have become more active since then. Could anybody explain that behaviour? --Chricho (talk) 00:53, 31 October 2012 (UTC)

I also noticed this error. Should be immediately fixed since it confuses users (including me) who got more active in the past half year and users who actually got less active in the past half year don't see the question "Why did you become less active on Wikipedia?". As a result we don't get their quite important feedback! -- Patrick87 (talk) 01:11, 31 October 2012 (UTC)

Yes. This. Also, for my situation in particular, I would have liked to see "were you more or less active on Wikipedia" three years ago, or five years ago. I was much more active from 2006-2008 than I have been since, and I have reasons for that. "Why did you become less active?" would be a great question. Frankly, I don't know what the Foundation can do about it, but I think that the culture of Wikipedia makes being an editor very frustrating. I ended up spending more time defending myself, and my authority, and trying to keep compromises and agreements established through debate and discussion intact (against attacks by new editors who weren't there when the compromise was made), rather than contributing to content. Editing Wikipedia should be fun, it should be relaxing. It should not be stressful, least of all should it be the greatest source of stress in my life. Like I said, I don't know what the Foundation can do about this, but I think it a key problem in the culture. Or, at least the culture as it was back in 2008. Now that I think about it, things might have changed dramatically already, it being a whole 4 years later... Still, that question on the survey "why are you less active today" would have been nice to see. LordAmeth (talk) 02:43, 31 October 2012 (UTC)

Translations are affected, too. --Chricho (talk) 01:39, 31 October 2012 (UTC)

Thanks, should be fixed now. Regards, Tbayer (WMF) (talk) 02:02, 31 October 2012 (UTC)

I had the same error when I did the survey a few minutes ago, using the German translation. It looks like it is NOT fixed yet. (Or maybe not fixed in every language?) Please correct that ASAP since - as Lexein points out further down - the current behaviour causes wrong raw data. -- Christallkeks (talk) 03:26, 31 October 2012 (UTC)

I have paused the banners for now while so we can sort this out. The survey flow (i.e. such conditional questions) should be the same for all translations. Obviously we may need to discard some data, but at least it should be restricted to this question. Still annoying of course. Thanks for your help. Regards, Tbayer (WMF) (talk) 03:40, 31 October 2012 (UTC)

I just did some research and found a possible origin for the bug: Counting Issues? The questionaire says:

12a. If you think back at half a year ago: [RANDOMIZE FIRST TWO QUESTIONS]
     a. Did you witness more or less conflict in the community than today?
     b. Did you feel more or less motivated to contribute to Wikipedia than today?
     c. (cf. Q7a) Were you more or less active on Wikipedia compared to now? [ANCHOR]

     more
     less
     same

[ASK ONLY IF Q7A=1]
12b. (Q7b) Why did you become less active on Wikipedia? Please choose all that apply.

Humans start counting with ONE -- answer[1]=more; answer[2]=less; answer[3]=same; -- which makes more the answer with the index number 1. But computers start counting at ZERO -- answer[0]=more; answer[1]=less; answer[2]=same; -- which makes less the answer with the index number 1. Basically ALL programming languages work in this way and possibly the survey software's markup language does so too. Maybe that's the bug's origin? -- Christallkeks (talk) 03:48, 31 October 2012 (UTC)

Just a minor point

You asked at the start of the survey which country the participant resides in but didn't include all countries in the drop down menu. For instance, I live in Bermuda and it's not there. If you include a drop down menu at the start of a survey that makes the participant feel unimportant or unnoticed they may not invest the time to complete it or think you don't want comments from people who reside in that country. Snowysusan (talk) 01:08, 31 October 2012 (UTC)

Good point. Thanks for pointing this out. We've added an "other" option at the bottom to include countries not on the list - User: Beckien (talk)

Education level

Hello. When they ask your education level in the French version, that's verry confusing because they use the terms in France but French is spoken in many countries that have different education systems. There should be a way to adapt it to your country or just describe them in a less confusing way. --174.89.214.125 02:46, 31 October 2012 (UTC)

Possible technical issues with the survey

I noticed two problems with the technical implementation of the survey.

1. The last page of the survey is a fake exit screen; it shows no actual questions but rather just the "Thank you very much [...]" text. However, it does have a next (>>) button leading to the actual exit screen – the same text in smaller print. I think many editors would mistakenly leave the survey site before actually submitting their responses. Will these "99% complete" survey forms be counted as fully complete ones?
2. I don't see an option to save the survey and continue it later, even though one is supposed to be there. Is there supposed to be some link users can bookmark? If there is, the instructions should mention it. PleaseStand (talk) 02:49, 31 October 2012 (UTC)

Me, too! I came here from what looked like the 2nd to last page. I can't get back there to complete. I hope the time I spent on the survey wasn't wasted!

Quick response regarding 1. (exit screen, see also [3]): This unfortunately a technical limitation of the survey software (Qualtrics) that already occurred in the previous survey and which we haven't been able to circumvent, even with the help of their tech support. The survey is intended to be left at the translated exit screen, and these "99% complete" questionnaires will be counted fully, as the translated exit/thank you message promises. Regards, Tbayer (WMF) (talk) 04:11, 31 October 2012 (UTC)

Sense error in question at 15%

"If you think back to half a year ago: "

"Were you more or less active on Wikipedia compared to now?".

I answered "Less", meaning that I was less active half a year ago. The next page asks,

"Why did you become less active on Wikipedia? Please choose all that apply. "

This is based on the opposite of my answer to the previous question; all the followup questions are based on becoming less active which I did not do. I think this is a showstopper error. There is no way for a user to know that the first answer is being recorded the way I clicked it, as a number, or as text (for "vote" validation), or if it is being recorded correctly, but interpreted incorrectly on all following pages.

As it is, this construction risks IMHO badly wrong raw data gathering. I'll cancel out and take the survey later, when the bugs are worked out.

• I see above that it was noted as fixed at 02:02 - it was not, since I checked in after that.

--Lexein (talk) 02:53, 31 October 2012 (UTC)

Thanks, we are looking into the reason why it's still not fixed yet. Regards, Tbayer (WMF) (talk) 03:22, 31 October 2012 (UTC)

Hi, just looked into this. Glad you caught it. It should work now. User: Beckien (talk) Beckien (talk) 06:40, 31 October 2012 (UTC)

It is still the same. Or do I have to start over to get a fixed version? --Chricho (talk) 08:31, 31 October 2012 (UTC)

See below -- I had the same problem at 11:15UTC.      Jim . . . . Jameslwoodward (talk to me) 11:31, 31 October 2012 (UTC)

Country List

The country where I live (Bhutan) is not available for selection in the "Country where you live" question. If all countries can't be listed at least include an "Other" option.

Thanks for pointing this out--an 'other' option has been added to the countries list User: Beckien (talk)

The country where I live is there, but incredibly hard to find: normally 日本語 is alphabetised as 'nihongo', but 日本 and 日本語 appeared somewhere among the i's and j's, and even some of the i's do not begin with i (!) -- Bahasa Indonesia, for example starts with 'B'. Really a selection list that long is unfriendly; at least allow keyboard entry of the two-letter codes as an option.

Imaginatorium (talk) 11:06, 31 October 2012 (UTC) (Japan)

Norwegian

There are 2 Norwegian language Wikipedias: norsk (nynorsk) and norsk (bokmål). On the lists for choosing language etc. I saw Bokmål twice and Nynorsk once. My preferred language is Nynorsk, but that choice wasn't an option. Please list BOTH of the Norwegian languages in these lists. Thank you.

This should be fixed now. Thanks for letting us know - User: Beckien (talk)

A few thoughts on the survey

1 - I felt uncomfortable answering the "Recruiting female editors" question. I fully support female editors, but I felt like any answer that I chose other than "yes" could be twisted into a dislike of female editors. I felt the issue was a non-issue, but selecting "Inappropriate subject for Wikipedia to be getting into" could be interpreted as a lot of things, depending upon the reader. I would have felt better with more information. Is there a shortage of female editors? Why is this question even being asked when anyone who wants to edit is free to register and edit? Is there something stopping female editors from joining and editing? I feel like I'm being asked my opinion on an argument I'm walking into with no background.

Hear hear!! Imaginatorium (talk) 11:08, 31 October 2012 (UTC)

2 - There's not much for folks who read Wikipedia often, but don't edit. All the questions were about editing. I only step in when I see something wrong, stated badly or grammatically incorrect. It doesn't happen much - hence the amount of time I spend editing Wikipedia: Low. The amount of time I spend on Wikipedia: A lot!

There's my two bits.--174.231.12.81 07:37, 31 October 2012 (UTC)

Interesting comments, thanks!

1. Gender gap and the links there might help to understand the background. Sorry that the question made you feel uncomfortable; having no opinion would have been a fine option as well.

2. Well yes, it's the "editors survey" ;) (The Wikimedia Foundation has also conducted a separate readers survey, which was advertised to all accessing Wikipedia, not just logged-in users as this survey.) That being said, it's useful to know how many people might log in mostly just to read, so your responses consist valuable information, too.

Regards, Tbayer (WMF) (talk) 07:58, 31 October 2012 (UTC)

German Version: describe the other writers with one word

This question is logically incorrect and therefore personally offending. There are some who are friendly and some who are not. So which word am I supposed to take?

"Uns interessiert, wie du deine Mitautoren beschreiben würdest. Unten findest du eine Liste von Wörtern, mit denen man Wikipedia-Autoren beschreiben kann. Bitte suche genau ZWEI Wörter aus, die am besten zur Beschreibung der Wikipedia-Autoren geeignet sind. Bitte wähle zwei." --Thomas Ledl (talk) 08:45, 31 October 2012 (UTC)

Further areas of exploration for the next survey

Firstly, one comment:

Frankly, I know next to nothing about the Foundation and, while I'm sure it does some good work, my answers were more or less "don't know"......

Main concern:

I think that the survey needs to obtain opinions from editors regarding their feelings about persistant vandalism, unregistered editors being allowed to contribute, etc. I spend far too much time reverting vandalism and reporting vandals (half of whom seem to be high school "students"). bUt, as soon as some are blocked, more appear.

While in general "good faith" may be assumed, it is very clear that obscenity and nonsence are deliberately added. There's got to be some sort of "three strikes and you're out" (to use an American baseball term) whereby vandals are blocked far more quickly. 78.145.29.209 09:43, 31 October 2012 (UTC)

• A need to rework the AGF policy has been my concern as well. We don't have to assume good faith; what we need instead is to not assume bad faith. The quote I submitted to the survey:

• We need to establish a better philosophy at Wikipedia. So far the AGF policy might look good, but it's unconvenient when it is applied to obvious spammers who have to get 3 warnings on their page before they're blocked - this is a waste of administrators effort. Similar applies to awkward attempts of established contributors to go yadda yadda I-think-you-did-it-in-good-faith for people who vandalise; there is a need for INDIVIDUAL ATTENTION to new editors instead of the wide-spread templates (ie warnings), so if they think that vandalise for a reason, it should be addressed lightly, in a human way and using discussion, rather than three kicks and a block. --Gryllida 10:53, 31 October 2012 (UTC)

• P.S. Yes, there is an option to input a text of one's liking ("what else could wikimedia foundation do to improve .." or something question text); it's not multiline but it's there and you can use it. --Gryllida 10:54, 31 October 2012 (UTC)

Some people don't understand what "movement" means

Some people told me that they are confused by the terms "Wikimedia Foundation" and "Wikimedia Movement". It sounded to them like it should be the same thing and they didn't understand why does it have a different name.

I guess that the survey should explain somewhere what is the difference between the Foundation, the chapters and the movement. If it's impossible to fix it in the current survey, then it should be noted for the next one. --Amir E. Aharoni (talk) 10:07, 31 October 2012 (UTC)

Lack of a generic feedback field.

Some Editors who are being surveyed probably have a long amount of text to contribute to explain what they think should be done to make Wikipedia better. However this survey lacks a generic multiline comments box. Adding it might be a good plan. --Gryllida 10:50, 31 October 2012 (UTC)

Two problems

At one point in the survey it asks if I was editing more, the same, or less than some time ago (I don't remember the time span). I checked "more". The next page then asked me to check reasons why I was editing less.

Also, the survey constantly referred to "Wikipedia" and did not include Commons in its choice lists. I took this to mean that I was supposed to answer only with respect to my activity on Wikipedia. Since I am an active Admin and CU on Commons, and spend most of my WMF time there, my answers were very different from what they might have been if I had included both.      Jim . . . . Jameslwoodward (talk to me) 11:28, 31 October 2012 (UTC)