View source for Learning and Evaluation/Archive/Program Evaluation and Design/Resources/Pilot Reporting Items

[[File:2013-06-22 10-37-05 Program Evaluation and Design Workshop Breakout Session 1 Headline Success 025.jpg|thumbnail|Icebreaker session about the positive outcomes of program evaluation in the Wikimedia movement at the first Program Evaluation workshop]]
==Your feedback is needed!==
Hi everyone! 

;The Program Evaluation & Design Team has been busy - thank you for your help so far in helping us better understand evaluation in the Wikimedia movement!;

We have been gathering information and feedback from program leaders through surveys and reaching out personally to some of you about [[Programs|Program Evaluation]]. Thank you for your participation and feedback thus far! It has been great for us to learn what is going on with these programs, their desired impact(s), and the types of data you’re gathering and/or looking to gather regarding evaluation of your programs.  

All of this information gathering has allowed the Program Evaluation & Design team to see what types of measures and data gathering goals program leaders, may be most capable of reporting and have in common across the programs we’ve been looking at since our team came together (Edit-a-thons, Editing workshops, GLAM content donations, Wiki Loves Monuments, Wiki Takes, WikiExpeditions, Wikipedia Education Program, on-wiki contests). 

;What’s happening next

In the upcoming week, we will invite program leaders to participate in reporting about their programs so that we may both test out this potential set of reporting items as well as begin systematic program evaluation efforts for this initial set of programs. Reporting this data will be voluntary, of course. 

These efforts will help a wide range of stakeholders (program leaders, grantmakers, and the movement at large), to better understand these programs and their impacts. Through this exploration we will begin to answer some of the key questions that we all have about these programs, their impact(s), and how they might choose programs that are effective toward these common programming goals (Participation, Content Production, Increasing Quality, Recruitment and Retention).

;Your feedback is critical 
 
We’d like community feedback on this set of questions and have created a document which details a question matrix. The question matrix features all of the potential reporting items that program leaders may be asked by program type. Specifically, each potential instruction and question on a separate row (or line) and a column for each program type which indicates whether the item will be included in that program’s reporting. For example, if you want to see what data will be requested about edit-a-thons, you’ll look at the column of questions and be able to know if it applies to edit-a-thon’s based on whether an “x” appears in the box for that question beneath the column heading “Edit-a-thons.” 

You can review it here:

https://docs.google.com/spreadsheet/ccc?key=0AqQxi0BdfXDsdDlRWHo0QjJWUUZVQWluSm4yWjBQTUE&usp=sharing 

Our apologies for the need to use a google spreadsheet, but as the matrix is rather complex, it is the best way we could share the full data and ensure its integrity. 

Please share your feedback on the bottom of this wiki page and make sure to reference any specific line number if you are providing feedback on a particular item. 

Think about things like language (if English is not your second language, are you comfortable with the wording?), question relevance, and specificity.
Please share your feedback with us by the '''end of day Thursday, September 26th''' as we hope to send survey invitations out by the end of the week.  

Thanks everyone!

Sarah, Jaime and Frank, 
[[Program Evaluation and Design]] team 

==Feedback==
Please share your feedback, with your signature, here!

===Gender and ethnicity?===

Interesting. WE will think on this and send considered responses.Nothing about ethnicity or gender? [[User:Jon Davies WMUK|Jon Davies WMUK]] ([[User talk:Jon Davies WMUK|talk]]) 08:01, 25 September 2013 (UTC)
:Thanks, Jon, we will look forward to your further feedback.  We have left an open essay reporting box for each of the four reporting areas (participation, content production, quality, and recruitment/retention) contained within the survey for people to report any additional data of interest that they happen to have (e.g. gender and diversity could be an observation included in that response category for participation and/or retention depending on what type of data one has collected). However, according to the evaluation capability status survey we collected last month, very few program leaders have been tracking that sort of demographic and diversity data up to this point.  Similar to targets for increased skill or interest which are also common, but currently absent, some target objectives will require survey inquiry or other further tool development in order for systematic collection and reporting.  Those potential measures still need additional evaluation capacity developed for the majority of program leaders and are not included for this initial reporting round. Still, we will encourage program leaders to share data on those additional goal areas that they have targeted and tracked data for their specific programming impacts beyond this basic set of counts. Hope that makes sense. [[User:JAnstee (WMF)|JAnstee (WMF)]] ([[User talk:JAnstee (WMF)|talk]]) 15:51, 25 September 2013 (UTC)
::JAnstee, thanks for this reply - I'm having trouble parsing it though. You say ''"Similar to, also absent but common, targets for increased skill or interest, some objectives will require survey inquiry, and those potential measures need additional evaluation capacity developed in order for systematic collection and reporting"'' - could you rephrase that sentence please? [[User:Richard Symonds (WMUK)|Richard Symonds (WMUK)]] ([[User talk:Richard Symonds (WMUK)|talk]]) 11:43, 26 September 2013 (UTC)
:::Revised, hope that helps [[User:JAnstee (WMF)|JAnstee (WMF)]] ([[User talk:JAnstee (WMF)|talk]]) 17:25, 26 September 2013 (UTC)

===More suggestions===
Perhaps some of these random items might be useful additions...
:*Was/Where is the Wikipedia page about the event/project (useful for lurkers and gawkers)
:*Was any outside media coverage generated - how can that be measured?
:*What was spent on pr 
:*Where there  '''in kind donation''' by GLAM 
:*Did GLAM-host do anything to cross promote
:*Was there swag or reward offered to participants

In the US, institutions understand/LOVE reporting, so if we as volunteers are armed with reporting tools we can showcase to our host/partners, the likelihood of Wikipedia events being taken more seriously, seriously increases ... (*_*) [[User:Bdcousineau|Bdcousineau]] ([[User talk:Bdcousineau|talk]]) 15:44, 25 September 2013 (UTC)
::Hi Bettina. These are good suggestions, however, we didn't see them as being common reporting items across the movement in the last survey we did (that you took). We'll be reporting our findings about that this week in a blog. I track a lot of these things, well, in my head, when I've worked with GLAMs, so I do think there are some good suggestions, however, like I said, we noticed they weren't that common. To remedy that, we're putting blank comment boxes for each type of program we're gathering data about (i.e. GLAM content donation, edit-a-thon, etc) so you can share data like this "Amount of in kind donation by GLAM - $500 worth of catering for edit-a-thon") by filling it in yourself. And, if we do see many people reporting your suggestions, then we will know we need to have them in the next data collection survey. There is one question about press: "This [Program Name] project/event  has blogs or other online information written to tell others about it (published by yourself or others)." where respondents share links or whatever they want to share related to that. I just track it on the event page or in my case study. Usually isn't not really overwhelming press wise, so I can maintain it easily. You can also use that section to share what you and the GLAM did to promote. 
::Jaime might have some more thoughts, but, similar to what she said above, if more people are saying they are reporting about in kind donations and swag, then we'll include it in the survey next. Do let us know if you see any concerns with the verbiage being used. Thanks for your thoughts so far! [[User:SarahStierch|SarahStierch]] ([[User talk:SarahStierch|talk]]) 16:23, 25 September 2013 (UTC)
::: Thank you for your input, Bettina! Yes, unfortunately, program level budget tracking is one of those areas that is weak currently. However, the evaluation questions related to your suggestions around promotion as well as budget allocations and resources are definitely something that will be included in any follow-up inquiry into any highly efficient or impactful programs that we identify through this initial process. Further, we are still considering whether including at least an indicator of budget categories may still be manageable at this phase (i.e., check box for in-kind donations, check box for whether awards were given) and which would be most informative for this purpose. [[User:JAnstee (WMF)|JAnstee (WMF)]] ([[User talk:JAnstee (WMF)|talk]]) 18:47, 26 September 2013 (UTC)

===Concerns===
Hi guys! Many thanks for doing this. We have not had time to construct any detailed feedback - indeed, I haven't had time or set up a proper meeting about this at WMUK. As a result, this is a hastily worded feedback note to say that I personally have a few concerns:
*You've asked for feedback on this report with a short turnaround time - just two days for feedback, right in the middle of the work we're doing for the FDC bid. We at Wikimedia UK really want to give some detailed feedback on this, but we don't have the staff time to do so if we are to get our bid in by Monday. I hope, therefore, that future feedback will have a later deadline - perhaps a fortnight, or a month. I know that this is a long turnaround, but we simply don't have the resources to write feedback at two day's notice.
*Secondly, we've chatted about this informally in the office, and are very concerned at the length of the "Data Report Request Item Matrix by Program". The sheer amount of feedback you are requesting will mean that you will only get feedback from larger chapters on their events - a volunteer will not fill in a form of this length. I am worried that by asking for people to fill in this form, you will not only get very little completed responses, but you will actively discourage volunteers and smaller thorgs from doing anything at all. I would recommend that you limit your feedback to, at most, 25% of the current length. 
*Don't apologise for using Google Spreadsheets - they are perfect in this situation, and good metrics are much more important than shoehorning things onto wikis.
*Try and simplify your language. Naming the spreadsheet 'Data Report Request Item Matrix by Program' rather than 'Feedback Questions' is a case in point! Row 28 in particular has extremely complex language: it scores just 19.5 on the Flesch Reading Ease test: it's about as complex to read as the Harvard Law Review. You should, I think, be aiming for a readability of about 50 or 60. 
I appreciate the complexity of today's reporting standards, but the spreadsheet and this page need to be much simplified if you are to have any real community input on these... I want this to succeed as much as you guys do :-) [[User:Richard Symonds (WMUK)|Richard Symonds (WMUK)]] ([[User talk:Richard Symonds (WMUK)|talk]]) 14:29, 27 September 2013 (UTC)
:Hi Richard! Thanks for your feedback. First, we understand it was a short period of time. We did promote it heavily to the community who is interested in evaluation and we've noticed a trend: we rarely receive feedback when we ask for it. So, that means we're doing something right, or people don't care ;)  (I'm joking, sort of!) But, in the future we will allow more time. Sadly, we're also under pressure to get this survey up and out in a specific period of time so we can have it for the community to look at before the end of November, or we would have had this up earlier or at a different more laid back time. Next time we do this, since this is the ''first time'' we'll provide more feedback time. Lesson learned!<br>
:You stated that only program leaders from larger chapters will respond - that's making an assumption, something I have learned is one of the most important things ''not'' to do in evaluation. When we release this survey, and we see in the next week and a  half that only large chapters have replied, then we will learn from that. But, perhaps that won't be the case. We were pretty amazed by the response to the last survey (learn more [http://blog.wikimedia.org/2013/09/27/survey-strong-culture-of-evaluation-room-for-growth/ here]) - we had a high reply rate with 32 chapters, three affiliates and eight individuals replying. We also learned that the majority of respondents had no current staff support in the work they were doing, meaning, voluntary. So, we'll see what the response is like post survey. But anyway.. :)
:I'm always paranoid, as someone who supports free and open software about using Google Docs in the movement. Glad it's OK with you!
:I agree also about the complexity of the language. We worked with multiple community members who use English as a second language to look at each question on the actual survey itself, and modified the language. I do agree the matrix thing is confusing, absolutely. It's a challenge for us to meet in the middle often, we don't want to dumb things down, but we also want to tone down the academic language. Do you have a link to an online tool that provides the score you produced? I'd love to use it in the future. Thanks for the feedback! This is the first time we've done this, so we'll see how it goes. Jaime might reply here shortly, too. [[User:SarahStierch|SarahStierch]] ([[User talk:SarahStierch|talk]]) 16:30, 27 September 2013 (UTC)
:: Thank you for sharing your concerns.  We realize that reporting is an additional burden and have designed the survey collector so that program leaders may report on as little as one program and programming event up to a handful.  Although some may not have time to share we are hoping enough will to allow us to try out these indicators and begin to surface data about the level of impact one might expect from each of these types of programs to help you all direct your programming to successfully meet your individual/chapter goals and targets. The most common and core targets for which program leaders reported already having tracked data are those which we have included at this time.  As we move toward systematic evaluation and developing guidance on metrics, we will need to proceed through several iterations and request program leader support in testing out these metrics by actually taking the time to pull the data, we appreciate all the support we have received so far and are optimistic about continued cooperation as we work together to define what self-evalaution will look like for Wikimedia programs. 
Thank you for catching that complex sentence (I broke it into two and that cell's text now reads at 50.4 for reading ease (although it now contains a passive sentence as well) - As Sarah said, we have had several non-native English speaking program leaders for the different programs review each of the actual items and have adjusted the language where we have identified points of confusion. Thank you for that additional flag on wording there.  If you have more specific points for language clarification, please let us know.
As for the timing, faced with our current reporting and decision-making demands we are on a very tight timeline for this first round of data collection and reporting. As we move forward and establish a more consistent reporting flow, we will try to ensure more review time when we request feedback. We apologize for the brief window, but we thought a brief window is better than none, and took the risk of this complaint. Thank you for taking the time that you have. [[User:JAnstee (WMF)|JAnstee (WMF)]] ([[User talk:JAnstee (WMF)|talk]]) 18:03, 27 September 2013 (UTC)

===Application===
I am wondering what is the intended application of this report? Is it listing suggestions for future metrics (and so we can choose the relevant ones that fit into our priorities and start recording now), or is it something we will be asked to complete in the coming weeks while looking retrospectively on past 12 months?  [[User:Daria Cybulska (WMUK)|Daria Cybulska (WMUK)]] ([[User talk:Daria Cybulska (WMUK)|talk]]) 16:17, 27 September 2013 (UTC)

:Yes, Daria, we are actually doing both.  Since so many program leaders reported having collected some of these more basic data points on their last year's programs, we are trying out some of these more basic metrics (where available and people are willing to share) to (1) see how well they help to inform us about the impacts that are commonly seen in each of these programs so that we may further prioritize for future reporting guidelines (this is only a beginning and only looking at a small set of potential metrics) and (2) begin to collect some normative data so that program leaders can compare their impacts to those that are seen more generally.  So these are also the actual measurements we will be requesting program leaders to report in the pending reporting request, however, that does not mean they are the final recommendations for regular reporting.  There is a new blog [[Wikimedia_Blog/Drafts/Wikimedia_community_evaluation_status_report_and_next_steps|blog]] about it. [[User:JAnstee (WMF)|JAnstee (WMF)]] ([[User talk:JAnstee (WMF)|talk]]) 17:46, 27 September 2013 (UTC)