2

The results of the second site self-evaluation have been published and some discussion of the detail commenced.

I wonder what purpose is intended to be served by this evaluation? Once we have settled on this, we will be in a position to discuss some methodological issues, such as:

  • How many respondents are required for the results to be useful?
  • Of whom should the respondents be representative?
  • How can we ensure that a "sufficient" number of the "right" people participate in the next survey?

But first things first -- what do we expect to get from this survey and what do we plan to do with it?

3
  • IMO the "right" people would be a cross-sample of all the active members of this site -- i.e. anyone who's ever posted or answered. But that's cloud-cuckoo land.
    – user104
    Commented Jun 13, 2013 at 9:37
  • My major concern about the methodology is that the instructions are so wide and open to interpretation. "How easily can a question be found with a search engine?" (I paraphrase). Well, given a question in front of me, I can construct a search that will find it 100% of the time. So that's good? If it's about a named ancestor, even a fuzzy search on parts of the ancestors details will probably find it. So that's good? But a search for the underlying more generally applicable info (say sources for State X in the period Y) doesn't show it up. On which basis do I assess it? Depends on my bias...
    – user104
    Commented Jun 13, 2013 at 9:42
  • As a relatively recent participant, what I'd like to see is feedback on how I can write a better question or answer. If my answer was satisfactory or needs improvement, how can I improve it? It ought to be clear by now that while I am hanging back in favor of the more experienced users in editing other people's questions, I am not shy about editing my own stuff.
    – Jan Murphy Mod
    Commented Mar 10, 2014 at 3:15

3 Answers 3

Reset to default
2

The purpose of the Site Self-Evaluation is to be a bit introspective — to step back and capture periodic snapshots of how the site looks from the outside (the Internet) looking in. It's easy to get caught up in the daily rigamarole of 'questions per day', reputation, and 'percent answered', but it's important on occasion to take a step back to see the forest for the trees — how is this site doing in this subject space?

I petitioned hard to keep this a site self-examination. The respondents should be representative of the broader community; as broad as possible. The more participants, the better.

Personally, I'm not nearly as interested in the number of 'better, worse, or same' ratings given, inasmuch as what communities DO with that information. What activities and discussion are generated around what you discovered, if anything? Are there broader issues that can be addressed? Or at the very very least, are the problems uncovered by the sample questions even fixed… or is the community generally content with the status quo?

No two community evaluations are alike, nor can they be used as a basis for comparison. It is impossible to assign an objective pass/fail grade based on the numerical results; and that is entirely besides the point. The purpose is to step back from all the analytics and take a look at the "essay portion" of the work you are doing here — how does the site perform from the outside looking in? Where do you fit in among the other sites covering this address space?

What you should expect to get from these surveys and what you plan to do with it is entirely up to you. Not to be too evasive, but that's sort of the point of a self-evaluation.

3
  • The point of a self-evaluation is that it is owned and directed by the participants. I had hoped that my question would move us toward that situation. What we have currently is a quasi-self-evaluation driven from Central Office.
    – Fortiter
    Commented Jun 13, 2013 at 4:21
  • I am struggling a little with the idea of an evaluation "from the outside ...looking in" when 100% of the data comes from those at the very bottom of the trenches.
    – Fortiter
    Commented Jun 13, 2013 at 4:24
  • @Fortiter, I think this is only a self-evaluation in that the assessment of the sample of content it is carried out by the participants, who also have the freedom to decide what if anything they wish to do right now about the results presented to them. It's hard to see how we can organise anything ourselves that's different but perhaps you have suggestions?
    – user104
    Commented Jun 13, 2013 at 9:35
1

Another slant on the purpose is in the podcast here: Podcast 48 (discussion starts at circa 40 minutes in).

For those who don't "do" podcasts, the key points that the site evaluation highlights seem to be:

  • What's the 'quality' trend on the site -- do our members think we're getting better, worse or holding our own?

  • How do we compare with the competition -- is our content better or worse than can be found elsewhere?

  • What do we as a community do about content that's evaluated as less good than we wish -- does anybody actually go and improve it (thanks Jeni for editing one of the Qs that needed attention already).

So, I think we need to do a comparison with our performance last time around to see the trend (in quality and participation in the evaluation); and go fix the questions we thought were lacking in some way; and ask ourselves which other sites have better content and why.

2
  • If each of the 10 questions (and their answers) selected for the evaluation were to be edited to remove perceived weaknesses, what impact would it have on overall site quality? If they were a truly representative sample and the opinions expressed were valid, the changes would be a drop in the bucket of what was (apparently) needed. If the methodology is flawed in either aspect, the beneficial impact will be even less.
    – Fortiter
    Commented Jun 13, 2013 at 4:34
  • @Fortier If the questions with negative scores in the evaluation were improved, and also other questions that individuals perceived to share the same weaknesses, it would be a first step. And if individuals learn from the site evaluation and feed that into constructing their own future questions and answers, or editing those posted by others to eliminate the same weaknesses, it would be a second step. Or we could do nothing. That's always an option if we're content with the status quo.
    – user104
    Commented Jun 13, 2013 at 9:03
0

Some context that might aid the discussion from: Which questions need Community Evaluaton?

Think of it as a periodic sanity check and a way to make sure that the site's still doing well. Are the questions of high quality? Are the answers better than what someone can easily find through Google?

and

It's an opportunity for the users to take a look at a sample of the questions and see if the site really is helping "make the Internet better" or if it falls short in some ways. Naturally, if it's the latter, the next step would be to get together and figure out how to correct the course.

and

For beta sites, they will, along with other metrics, also help us on the community team to gauge whether the site is ready to be self-sustaining and helpful for a long time

1
  • 1
    A great start to the conversation BUT the question on SO that is linked is highly misleading (although the answer is useful). It is not the question that needs evaluation but the site (an aggregation of questions, comments and answers). A very important difference.
    – Fortiter
    Commented Jun 12, 2013 at 12:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .