Wikipedia:Wikipedia Signpost/2023-10-03/Recent research

Recent research

Concerns about limiting Wikipedia's "anyone can edit" principle "may be overstated"

By Tilman Bayer

A monthly overview of recent academic research about Wikipedia and other Wikimedia projects, also published as the Wikimedia Research Newsletter.

FlaggedRevs study finds that concerns about limiting Wikipedia's "anyone can edit" principle "may be overstated"

A paper titled "The Risks, Benefits, and Consequences of Prepublication Moderation: Evidence from 17 Wikipedia Language Editions", from last year's CSCW conference, addresses a longstanding open question in Wikipedia research, with important implications for some current issues.

Wikipedia famously allows anyone to edit, which generally means that even unregistered editors can make changes to content that go live immediately - only subject to "postpublication moderation" by other editors afterwards. Less well known is that on many Wikipedia language versions, this principle has long been limited by a software feature called Flagged Revisions (FlaggedRevs), which was developed and designed on the request of German Wikipedia community and deployed there first in 2008, and has since been adopted by various other Wikimedia projects. (These do not include the English Wikipedia, which after much discussion implemented a system called "Pending Changes" that is very similar, but is only applied on a case-by-case basis to a small percentage of pages.) As summarized by the authors:

FlaggedRevs is a prepublication content moderation system in that it will display the most recent “flagged” revision of any page for which FlaggedRevs is enabled instead of the most recent revision in general. FlaggedRevs is designed to “give additional information regarding quality,” by ensuring that revisions from less-trusted users are vetted for vandalism or substandard content (e.g., obvious mistakes because of sloppy editing) before being flagged and made public. The FlaggedRevs system also displays the moderation status of the contribution to readers. [...] Although there are many details that can vary based on the way that the system is configured, FlaggedRevs has typically been deployed in the following way on Wikipedia language editions. First, users are divided into groups of trusted and untrusted users. Untrusted users typically include all users without accounts as well as users who have created accounts recently and/or contributed very little. Although editors without accounts remain untrusted indefinitely, editors with accounts are automatically promoted to trusted status when they clear certain thresholds determined by each language community. For example, German Wikipedia automatically promotes editors with accounts who have contributed at least 300 revisions accompanied by at least 30 comments.

The paper studies the impact of the introduction of FlaggedRevs "on 17 Wikipedia language communities: Albanian, Arabic, Belarusian, Bengali, Bosnian, Esperanto, Persian, Finnish, Georgian, German, Hungarian, Indonesian, Interlingua, Macedonian, Polish, Russian, and Turkish" (leaving out a few non-Wikipedia sister projects that also use the system). The overall findings are that

"the system is very effective at blocking low-quality contributions from ever being visible. In analyzing its side effects, we found, contrary to expectations and most of our hypotheses, little evidence that the system neither raises transaction costs sufficiently to inhibit participation by the community as a whole, nor measurably improves the quality of contributions."

In the "Discussion" section, the authors write

Our results suggest that prepublication moderation systems like FlaggedRevs may have a substantial upside with relatively little downside. If this is true, why are a tiny proportion of Wikipedia language editions using it? Were they just waiting for an analysis like ours? In designing this study, we carefully read the Talk page of FlaggedRevs. Community members commenting in the discussion agreed that prepublication review significantly reduces the chance of letting harmful content slip through and being displayed to the public. Certainly, many agreed that the implementation of prepublication review was a success story in general—especially on German Wikipedia. [...]
However, the same discussion also reveals that the success of German Wikipedia is not enough to convince more wikis to follow in their footsteps. From a technical perspective, FlaggedRevs’ source code appears poorly maintained. [...] FlaggedRevs itself suffers from a range of specific limitations. For example, the FlaggedRevs system does not notify editors that their contribution has been rejected or approved. [...] Since April 2017, requests for deployment of the system by other wikis have been paused by the Wikimedia Foundation indefinitely. Despite these problems, our findings suggest that the system kept low-quality contributions out of the public eye and did not deter contributions from the majority of new and existing users. Our work suggests that systems like FlaggedRevs deserve more attention.

(This reviewer agrees on particular regarding the lack of notifications for new and unregistered editors that their edit has been approved - having filed, in vain, a proposal to implement this uncontroversially beneficial and already designed software feature to the annual "Community Wishlist", in 2023, 2022, and 2019.)

Interestingly, while the FlaggedRevs feature was (as summarized by the authors) developed by the the Wikimedia Foundation and the German Wikimedia chapter (Wikimedia Deutschland), community complaints about a lack of support from the Foundation for the system were present even then, e.g. in a talk at Wikimania 2008 (notes, video recording) by User:P. Birken, a main driving force behind the project. Perhaps relatedly, the authors of the present study highlight a lack of researcher attention:

"Despite its importance and deployment in a number of large Wikipedia communities, very little is known regarding the effectiveness of the system and its impact. A report made by the members of the Wikimedia Foundation in 2008 gave a brief overview of the extension, its capabilities and deployment status at the time, but acknowledged that “it is not yet fully understood what the impact of the implementation of FlaggedRevs has been on the number of contributions by new users.” Our work seeks to address this empirical gap."

Still, it may be worth mentioning that there have been at least two preceding attempts to study this question (neither of these has been published in peer-reviewed form, thus their omission from the present study is understandable). They likewise don't seem to have identified major concerns that FlaggedRevs might contribute to community decline:

1. A talk at Wikimania 2010 presented preliminary results from a study commissioned by Wikimedia Germany, e.g. that on German Wikipedia, "In general, flagged revisions did not [affect] anonymous editing" and that "most revisions got approved very rapidly" (the latter result surely doesn't hold everywhere; e.g. on Russian Wikipedia, the median time for an unregistered editor's edit to get reviewed is over 13 days at the time of writing). It also found, unsurprisingly, a "reduced impact of vandalism", consistent with the present study.
2. An informal investigation of an experiment conducted by the Hungarian Wikipedia in 2018/19 similarly found that FlaggedRevs had "little impact on the growth of the editor community" overall. The experiment consisted of deactivating the feature of FlaggedRevs that hides unreviewed revisions from readers. As a second question, the Hungarian Wikipedians asked "How much extra load does [deactivating FlaggedRevs] put on patrollers?" They found that "[t]he ratio of bad faith or damaging edits grew minimally (2-3 percentage points); presumably it is a positive feedback for vandals that they see their edits show up publicly. The absolute number of such edits grew significantly more than that, since the number of anonymous edits grew [...]."

In any case, the CSCW paper reviewed here presents a much more comprehensive and methodical approach, not just in studying the impact of FlaggedRevs across multiple wikis, but also regarding the formalizing of various research hypotheses and concerning the use of more reliable statistical techniques.

(Disclosure: This reviewer provided some input to the authors at the beginning of their research project, but was not involved in it otherwise.)

See also related earlier coverage: "Sociological analysis of debates about flagged revisions in the English, German and French Wikipedias" (2012)

Briefly

• Wikimania, the annual global conference of the Wikimedia movement, took place in Singapore in August (as an in-person event again for the first time since 2019). Its research track included the by now traditional State of Wikimedia Research presentation highlighting research trends from the past year (with involvement by members of this research newsletter).
• See the page of the monthly Wikimedia Research Showcase for videos and slides of past presentations.

Other recent publications

Other recent publications that could not be covered in time for this issue include the items listed below. Contributions, whether reviewing or summarizing newly published research, are always welcome.

"Wikidata as Semantic Infrastructure: Knowledge Representation, Data Labor, and Truth in a More-Than-Technical Project"

From the abstract:

"Various Wikipedia researchers have commended Wikidata for its collaborative nature and liberatory potential, yet less attention has been paid to the social and political implications of Wikidata. This article aims to advance work in this context by introducing the concept of semantic infrastructure and outlining how Wikidata’s role as semantic infrastructure is the primary vehicle by which Wikipedia has become infrastructural for digital platforms. We develop two key themes that build on questions of power that arise in infrastructure studies and apply to Wikidata: knowledge representation and data labor."

"Naked data: curating Wikidata as an artistic medium to interpret prehistoric figurines"

From the abstract:

"In 2019, Digital Curation Lab Director Toni Sant and the artist Enrique Tabone started collaborating on a research project exploring the visualization of specific data sets through Wikidata for artistic practice. An art installation called Naked Data was developed from this collaboration and exhibited at the Stanley Picker Gallery in Kingson, London, during the DRHA 2022 conference. [...] This article outlines the key elements involved in this practice-based research work and shares the artistic process involving the visualizing of the scientific data with special attention to the aesthetic qualities afforded by this technological engagement."

"What makes Individual I's a Collective We; Coordination mechanisms & costs"

From the abstract:

"Diving into the Wikipedia ecosystem [...] we identified and quantified three fundamental coordination mechanisms and found they scale with an influx of contributors in a remarkably systemic way over three order of magnitudes. Firstly, we have found a super-linear growth in mutual adjustments (scaling exponent: 1.3), manifested through extensive discussions and activity reversals. Secondly, the increase in direct supervision (scaling exponent: 0.9), as represented by the administrators’ activities, is disproportionately limited. Finally, the rate of rule enforcement exhibits the slowest escalation (scaling exponent 0.7), reflected by automated bots. The observed scaling exponents are notably robust across topical categories with minor variations attributed to the topic complication. Our findings suggest that as more people contribute to a project, a self-regulating ecosystem incurs faster mutual adjustments than direct supervision and rule enforcement."

"Wikidata Research Articles Dataset"

From the abstract:

"The "Wikidata Research Articles Dataset" comprises peer-reviewed full research papers about Wikidata from its first decade of existence (2012-2022). This dataset was curated to provide insights into the research focus of Wikidata, identify any gaps, and highlight the institutions actively involved in researching Wikidata."

References

Supplementary references and notes:

This page is a draft for the next issue of the Signpost. Below is some helpful code that will help you write and format a Signpost draft. If it's blank, you can fill out a template by copy-pasting this in and pressing 'publish changes': {{subst:Wikipedia:Wikipedia Signpost/Templates/Story-preload}}

Images and Galleries

Sidebar images To put an image in your article, use the following template (link): [[File:|center|300px|alt=TKTK]] O frabjous day. {{Wikipedia:Wikipedia Signpost/Templates/Filler image-v2 |image = |size = 300px |alt = TKTK |caption = |fullwidth = no }} This will create the file on the right. Keep the 300px in most cases. If writing a 'full width' article, change |fullwidth=no to |fullwidth=yes. Inline images Placing {{Wikipedia:Wikipedia Signpost/Templates/Inline image |image = |size = 300px |align = center |alt = Placeholder alt text |caption = CAPTION }} (link) will instead create an inline image like below [[File:|300px|center|alt=Placeholder alt text]] CAPTION Galleries To create a gallery, use the following <gallery mode = packed | heights = 200px> |Caption for second image </gallery> to create

Quotes

Framed quotes “ Lorem ipsum dolor sit amet... ” — AUTHOR, SOURCE To insert a framed quote like the one on the right, use this template (link): {{Wikipedia:Wikipedia Signpost/Templates/Filler quote-v2 |1 = |author = |source = |fullwidth = }} If writing a 'full width' article, change |fullwidth=no to |fullwidth=yes. Pull quotes To insert a pull quote like “ Lorem ipsum dolor sit amet... ” use this template (link): {{Wikipedia:Wikipedia Signpost/Templates/Quote |1 = |source = }} Long quotes To insert a long inline quote like The goose is on the loose! The geese are on the lease! — User:Oscar Wilde — Quotations Notes from the Underpoop use this template (link): {{Wikipedia:Wikipedia Signpost/Templates/block quote | text = | by = | source = | ts = | oldid = }}

Side frames

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. A caption Side frames help put content in sidebar vignettes. For instance, this one (link): {{Wikipedia:Wikipedia Signpost/Templates/Filler frame-v2 |1 = Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. |caption = A caption |fullwidth = no }} gives the frame on the right. This is useful when you want to insert non-standard images, quotes, graphs, and the like. Example − Graph/Charts Further information: Category:Graph, chart and plot templates A caption For example, to insert the {{Graph:Chart}} generated by {{Graph:Chart |width=250|height=100|type=line |x=1,2,3,4,5,6,7,8|y=10,12,6,14,2,10,7,9 }} in a frame, simple put the graph code in |1= {{Wikipedia:Wikipedia Signpost/Templates/Filler frame-v2 |1= {{Graph:Chart |width=250|height=100|type=line |x=1,2,3,4,5,6,7,8|y=10,12,6,14,2,10,7,9 }} |caption=A caption |fullwidth=no }} to get the framed Graph:Chart on the right. If writing a 'full width' article, change |fullwidth=no to |fullwidth=yes.

Two-column vs full width styles

If you keep the 'normal' preloaded draft and work from there, you will be using the two-column style. This is perfectly fine in most cases and you don't need to do anything. However, every time you have a |fullwidth=no and change it to |fullwidth=yes (or vice-versa), the article will take that style from that point onwards (|fullwidth=yes → full width, |fullwidth=no → two-column). By default, omitting |fullwidth= is the same as putting |fullwidth=no and the article will have two columns after that. Again, this is perfectly fine in most cases, and you don't need to do anything. However, you can also fine-tune which style is used at which point in an article. To switch from two-column → full width style midway in an article, insert {{Wikipedia:Wikipedia Signpost/Templates/Signpost-block-end-v2}} {{Wikipedia:Wikipedia Signpost/Templates/Signpost-block-start-v2|fullwidth=yes}} where you want the switch to happen. To switch from full width → two-column style midway in an article, insert {{Wikipedia:Wikipedia Signpost/Templates/Signpost-block-end-v2}} {{Wikipedia:Wikipedia Signpost/Templates/Signpost-block-start-v2|fullwidth=no}} where you want the switch to happen.

Article series

To add a series of 'related articles' your article, use the following code Related articles Visual Editor Five, ten, and fifteen years ago 1 January 2023 VisualEditor, endowment, science, and news in brief 5 August 2015 HTTPS-only rollout completed, proposal to enable VisualEditor for new accounts 17 June 2015 VisualEditor and MediaWiki updates 29 April 2015 Security issue fixed; VisualEditor changes 4 February 2015 More articles Draft namespace, VisualEditor meetings 25 December 2013 Sockpuppet investigations, VisualEditor, Wikidata's birthday, and more 6 November 2013 Last call for Wiki Loves Monuments; Community–WMF tension over VisualEditor 25 September 2013 Making Wikipedia more accessible 4 September 2013 Gallery improvements launch on Wikipedia 28 August 2013 The VisualEditor Beta and the path to change 31 July 2013 Defining consensus; VisualEditor default state; expert and layperson terms in article titles 31 July 2013 VisualEditor in midst of game-changing deployment series 3 July 2013 VisualEditor will "change world history" 12 June 2013 VisualEditor, elections, bots, and more 12 June 2013 The Visual Editor: Where are we now, and where are we headed? 25 March 2013 Visual Editor "on schedule" for July rollout 18 March 2013 Looking ahead to 2013 7 January 2013 The new Visual Editor gets a bit more visual 10 December 2012 Second Visual Editor prototype launches 25 June 2012 Visual editor demo launched, hailed as "most important change to our user experience ... ever"; but elsewhere over-hasty deployments criticised 19 December 2011 Full steam ahead on Visual Editor, the avoidance of lock-breaking and 1.18b1 release 7 November 2011 {{Signpost series |type = sidebar-v2 |tag = VisualEditor |seriestitle = Visual Editor |fullwidth = no }} or {{Signpost series |type = sidebar-v2 |tag = VisualEditor |seriestitle = Visual Editor |fullwidth = yes }} will create the sidebar on the right. If writing a 'full width' article, change |fullwidth=no to |fullwidth=yes. A partial list of valid |tag= parameters can be found at here and will decide the list of articles presented. |seriestitle= is the title that will appear below 'Related articles' in the box. Alternatively, you can use {{Signpost series |type = inline |tag = VisualEditor |tag_name = visual editor |tag_pretext = the }} at the end of an article to create For more Signpost coverage on the visual editor see our visual editor series. If you think a topic would make a good series, but you don't see a tag for it, or that all the articles in a series seem 'old', ask for help at the WT:NEWSROOM. Many more tags exist, but they haven't been documented yet.

Links and such

This department (Recent research): Next issue · Next next issue · Search for previous Arbitration report: Arbitration · Arbcom · Archives · Cases · Enforcements Discussion report: News and notes: In the media: Google News · TechMEME By the way, the template that you're reading right now is {{Editnotices/Group/Wikipedia:Wikipedia Signpost/Next issue}} (edit). A list of the preload templates for Signpost articles can be found here.

← Previous "Recent research"

In this issue

3 October 2023

News and notes

In the media

Recent research

Featured content

Concept

Poetry

Traffic report

+ Add a comment

Discuss this story

These comments are automatically transcluded from this article's talk page. To follow comments, add the page to your watchlist. If your comment has not appeared here, you can try .

FlaggedRevs

There seem to be some errors in The Risks, Benefits, and Consequences of Prepublication Moderation: Evidence from 17 Wikipedia Language Editions (https://arxiv.org/abs/2202.05548) papers assumptions on how FlaggedRevs works. For example:

1. ... FlaggedRevs is a prepublication content moderation system in that it will display the most recent “flagged” revision of any page for which FlaggedRevs is enabled instead of the most recent revision in general... - This is depending on wiki where user is. For example German Wikipedia uses pre-moderation, and as different setup example is Finnish Wikipedia which uses post-moderation.
2. ... Although editors without accounts remain untrusted indefinitely, editors with accounts are automatically promoted to trusted status when they clear certain thresholds determined by each language community.... - This also depends on wiki. For example dewiki promotes automatically, but huwiki has rather small number manually selected trusted reviewers. Fiwiki has large number manually selected trusted reviewers.

--Zache (talk) 09:11, 4 October 2023 (UTC)

In Russian Wikipedia, as well as in Russian Wikinews, FlaggedRevs is a disaster. You say Germans are guilty in that? --ssr (talk) 06:09, 27 October 2023 (UTC)

Wikidata

Wikidata as Semantic Infrastructure: Knowledge Representation, Data Labor, and Truth in a More-Than-Technical Project.

  The article has a lot of critiques of Wikidata, some interesting, some less relevant in my opinion. Some of them were a big part of the discussions around adopting Wikidata on frwiki, interestingly, so might be a little bit out of date. The copyright issues, for example, was and is still I guess a show stopper for some. Interestingly I think this point is significantly weakened with the emergence of the LLMs like ChatGPT as a major products of the Gafams : they don’t really care about the copyright issues or the attribution of datas they learn from, and they really can’t track back the generated factoids origin or reliably source anything … Without any restraint to date from copyright laws. Copyright does not seem to slow automated fact extraction from any public source in any way … (for the rest of the arguments covered for or against Wikidata on wikipedias, this has already been extensively discussed and argued on the projects, I won’t return on this because I don’t really feel like it)

I don’t quite understand the upper ontology part, as I think Wikidata as is is difficult to really use as an upper ontology because of the lack of formalisation. There is no reasoning rules in Wikidata to capture some of the meaning, and this is a crucial part of how an upper ontology is supposed to be used. The upper tree is indeed, or as long been as there is ongoing effort to clear this up, globally a hard to manage mess. But I doubt anyone actually used in a meaningful way the classes they show up like bad, or actually used Wikidata as an upper ontology. I’d like this claim of the article to be given more substance or proof.

It’s a critic I would actually make for the whole article, this is a discussion of the critiques but there is not really a lot of datas to give substance, this is mostly qualitative, and as such arguable. TomT0m (talk) 11:38, 4 October 2023 (UTC)

ChatGPT v. Wikipedia

The study authors comment on prose quality. I happened to ask ChatGPT yesterday to explain what government shutdowns in the U.S. are and what effects they have. I got the following answer:

Government shutdowns in the United States occur when Congress fails to pass a budget or a continuing resolution to fund the government. This can happen due to disagreements between political parties or issues related to specific policies. The shutdown results in the temporary suspension of non-essential government services and the furlough of federal employees.

During a government shutdown, essential services such as national security, law enforcement, and emergency activities continue to operate. However, many government functions, agencies, and national parks may be affected. Federal employees who are deemed non-essential may be furloughed, meaning they are placed on temporary unpaid leave.

The economic impact can be significant, affecting various sectors, including tourism, government contractors, and the overall economy. Additionally, government shutdowns can lead to a loss of public confidence in the government's ability to function effectively.

It's worth noting that my information is up to September 2021, and I recommend checking the latest sources for updates on this topic.

I then compared that to the lead of Government shutdowns in the United States:

In the United States, government shutdowns occur when funding legislation required to finance the federal government is not enacted before the next fiscal year begins. In a shutdown, the federal government curtails agency activities and services, ceases non-essential operations, furloughs non-essential workers, and retains only essential employees in departments that protect human life or property. Shutdowns can also disrupt state, territorial, and local levels of government.

Funding gaps began to lead to shutdowns in 1980, when Attorney General Benjamin Civiletti issued a legal opinion requiring it. This opinion was not consistently adhered to through the 1980s, but since 1990 all funding gaps lasting longer than a few hours have led to a shutdown. As of September 2023, ten funding shutdowns have led to federal employees being furloughed.

The most significant shutdowns have included the 21-day shutdown of 1995–1996, during the Bill Clinton administration, over opposition to major spending cuts; the 16-day shutdown in 2013, during the Barack Obama administration, caused by a dispute over implementation of the Affordable Care Act (ACA); and the longest, the 35-day shutdown of 2018–2019, during the Donald Trump administration, caused by a dispute over funding an expansion of barriers on the U.S.–Mexico border.

Shutdowns disrupt government services and programs; they close national parks and institutions. They reduce government revenue because fees are lost while at least some furloughed employees receive back pay. They reduce economic growth. During the 2013 shutdown, Standard & Poor's, the financial ratings agency, said on October 16 that the shutdown had "to date taken $24 billion out of the economy", and "shaved at least 0.6 percent off annualized fourth-quarter 2013 GDP growth".

Personally I found ChatGPT's output a lot more readable than the Wikipedia lead – it is just better written. The English Wikipedia text often required me to go back and read the sentence again.

Take the first sentence: In the United States, government shutdowns occur when funding legislation ... At first I parsed "when funding legislation" as an indication of when shutdowns occur (i.e. "when you are funding legislation"). I needed to read on to realise that this wasn't where the sentence was going.

Next, Wikipedia uses the rather technical expression "when funding legislation ... is not enacted" (which is also passive voice) where ChatGPT uses the much easier-to-understand "when Congress fails to pass a budget" (active voice).

Where ChatGPT speaks of a "temporary suspension of non-essential government services", Wikipedia says the federal government "curtails agency activities and services, ceases non-essential operations", etc. I find the ChatGPT phrase easier to understand and faster to read while providing much the same information as the quoted Wikipedia passage (a point the study authors commented on specifically).

The Wikipedia sentence Funding gaps began to lead to shutdowns in 1980, when Attorney General Benjamin Civiletti issued a legal opinion requiring it. leaves me wondering even now what the word "it" at the end of the sentence is meant to refer to.

I suspect our sentence construction and word use are not helping us win friends. It's one thing when we are the only service available; it's another when there is a new kid on the block. Andreas JN466 13:56, 4 October 2023 (UTC)

I agree that ChatGPT produces better prose than most humans do (myself included), and as far as reliability is concerned, I'd honestly trust it to give me a general overview on any non-controversial topic, no matter how obscure (I tested it earlier on "the Black sermonic tradition of whooping" and it passed with flying colours). I think where it fails and starts to hallucinate is when you ask it to drill down into the detail. This is Wikipedia's strength – we may not be good at summarizing complex topics, but we're very good at detail.

So I don't think it has to be "machines vs. Wikipedians". I used ChatGPT this morning to write a lead for the article Khlysts. As leads go, it's maybe not perfect, but it's decent. But when I ask the bot for more in-depth information, like "Who founded the Khlysts?", it starts to get vague, and if I press it for specific answers, that's when it starts lying rather than admit it doesn't know. Chatbots have got a long way to go before they can write entire articles (at least without a lot of coaching and cross-checking, at which point you're not really saving any time), and I personally don't believe that AI will ever reach that level. But I can easily foresee a not-too-distant-future in which Wikipedia has an integrated LLM interface that allows editors to auto-generate summary-style content, or perhaps provides it to readers on the fly via a "Summarize this article/section" button.

In my view, the alarm of the study authors is misplaced. They seem to agree that the AI-generated content they used in the study is, in fact, trustworthy and competent, so there's no reason to be "unsettled" that people rate it as such. I suspect the participants would give a different rating to the vague, generic answers that ChatGPT gives you when you try to go beyond a summary. I don't think there's any competition here – when people want pedantic, granular detail, they'll always know where to find it. Sojourner in the earth (talk) 21:52, 4 October 2023 (UTC)

The fact that ChatGPT doesn't know when it doesn't know something means it's "intelligent" only in a restricted sense. To go on and on about a topic, all the while presenting itself as an "intelligence", also does not seem ethical. CurryCity (talk) 08:29, 5 October 2023 (UTC)

Quoted from above, "I'd honestly trust it to give me a general overview on any non-controversial topic, no matter how obscure" — that's extremely problematic. Recently I read about someone who is a critic with a dim view of a certain thing, a fact that is well known in his field (albeit not by the general public), that when he asked ChatGPT who he is, it smoothly and confidently claimed that he is a proponent of it. The devil is in details of the black box of "where it fails and starts to hallucinate is when you ask it to drill down into the detail" — but clearly the upshot is that you absolutely cannot trust GPT not to mix any bits of plausible-sounding-but-definitely-incorrect hallucination into any kind of "explanation" that it gives you about what something or someone is or isn't, when that something or someone is any more obscure than extremely widely known and understood. When I read in this Signpost article the sentence, " […] even if the participants know that the texts are from ChatGPT, they consider them to be as credible as human-generated and curated texts [from Wikipedia]," the first thought that popped into my head is, "Well, that's it, we're boned." Duck and cover and kiss our present standard of living goodbye, because some bad shit will be happening if that shit continues very long in its current configuration. You know how McNamara said that "the indefinite combination of human fallibility and nuclear weapons will destroy nations"? Like it doesn't even matter if the annualized risk is somewhat low, if you run the runtime for enough years? That's most likely what's going to happen with the combination of human [epistemologic] fallibility [incompetence] plus LLMs, unless the "indefinite" part can be massively shortened in duration by chaining today's type of LLMs in series with some kind of downstream rapid-acting bullshit-detector software that can act like a filter. If that shit doesn't get invented within the next N years, where time N is pretty fucking short, then there is going to come a time in the future where some Homer Simpson who works at a nuclear plant is going to ask an LLM how to run the reactor properly and the LLM is going to reply with some super-smoothly-worded and speciously/superficially-plausible-but-yet-idiotically-wrong answer, and there's going to be another Chernobyl-type steam explosion of a reactor, because even though the plant has a strict rule against asking any LLMs for any cheatsheet help, Homer is going to carry his smartphone into the bathroom and ask it while inside the toilet stall because he thinks he's getting away with a clever clandestine lifehack and sticking it to The Man. Or some librarian is going to ask an LLM which library books to ban for age-inappropriate content and it's totally going to confabulate a bunch of imaginary but plausible-sounding horseshit and get books banned for containing things that they don't even contain at all (that one already happened, by the way). Or a lawyer is going to ask an LLM to write a legal argument and cite a bunch of 'definitely real and please not fake' references that support it, and he's going to get fired because those smoothly cited plausible-sounding references are all made-up imaginary horseshit anyway (that one already happened, by the way). "Oh, no, don't be silly, Quercus, that reactor-explosion thing's not gonna happen." Yeah. We all sure better just hope not, right? Running on hope and faith but no guardrails at all, what could go wrong, right? This is why the likes of Sam Altman are begging governments to regulate their industry (albeit begging with crocodile tears in their eyes while they shovel wads of money into their bank accounts). Also, just to clear up a misconception implied on this page, among some of these other comments — regarding the notion that, "Well, how can you blame people, because a supposed-to-be-nonfiction answer that's factually wrong but super-smoothly worded and easy to read is preferable to one that's factually correct but somewhat clunkily worded." Jesus, Lord help us with the stupidity. If you have to ask what's wrong with that notion, then goto Homer's steam explosion above. Quercus solaris (talk) 23:37, 6 October 2023 (UTC)

Thing is, there are plenty of Wikipedia articles that also contain "plausible-sounding-but-definitely-incorrect" information, so for LLMs and Wikipedia to be rated as equally credible sounds about right. Of course I wouldn't use either as a source if I were writing my own article, but for satisfaction of personal curiosity, sure. As for the rest of your comment, I'm not convinced that "human fallibility plus LLMs" is a bigger danger to society than, say, human fallibility plus the invention of the printing press, or human fallibility plus firelighters. Pretty much anything that benefits humanity can be misused, and every Next Big Thing gets people prophesying the imminent collapse of civilization, but somehow we keep trucking along. Sojourner in the earth (talk) 05:40, 7 October 2023 (UTC)

In this case success and failure are asymmetric. It takes only one bad enough catastrophe once to wipe out civilisation, if not humanity. Even ones that are "too small" to destroy the rest of the biosphere or planet with us. CurryCity (talk) 06:33, 10 October 2023 (UTC)

Even if ChatGPT or its successor becomes the predominant internet search tool, that doesn't mean Wikipedia will be obsolete. It likely means that Wikipedia will go back to its theoretical origin as a reference work rather than the internet search tool many readers use it as. Thebiguglyalien (talk) 16:11, 4 October 2023 (UTC)

Ah, the rise of AI. I've used it to get ideas for small projects in the past, but people prefer LLMs over Wikipedia? That's, just... sad. The Master of Hedgehogs is back again! 22:09, 4 October 2023 (UTC)

• It's quite understandable because articles often have a ponderous, pedantic style. As a fresh example, consider the current FA, ministerial by-election. Its first sentence is

  From 1708 to 1926, members of parliament (MPs) of the House of Commons of Great Britain (and later the United Kingdom) automatically vacated their seats when made ministers in government and had to successfully contest a by-election in order to rejoin the House; such ministerial by-elections were imported into the constitutions of several colonies of the British Empire, where they were likewise all abolished by the mid-20th century.

  When this was a candidate, a reviewer observed that

  ... you have a tendency to write very long sentences, exemplified by the lead sentence, which is 67 words long. A quick google search tells me 25 words is the optimal sentence length (and 30 the maximum), but here you can routinely see double that.

  By my count, this sentence now has 68 words so it's only getting longer!

Andrew🐉(talk) 11:47, 5 October 2023 (UTC)

I really hope we can get a guideline on readability. My young sister-in-law complains that we adults "Explain things like Wikipedia", rather than in a way that a teenager can understand. I started to draft readability guideline once, but never found time to get it past the first couple of sentences. —Femke 🐦 (talk) 19:59, 6 October 2023 (UTC)

There are plenty of guidelines such as WP:NOTJARGON, WP:NOTJOURNAL and MOS:LEAD. The challenge is getting them followed. There are also plenty of metrics for this such as the automated readability index. We just need a bot or tool to assess all our articles and so highlight cases in need of attention. Andrew🐉(talk) 22:46, 6 October 2023 (UTC)

What is missing from these guidelines is a) the need to make non-technical text easy to understand b) more practical advice or what sentence and paragraph lengths make text difficult. I've been playing with the idea of creating a contest, similar to WP:the core contest, to tackle lead difficulty. Not sure yet what the best set-up is, as misguided attempts at simplifying can be quite bad. —Femke 🐦 (talk) 06:50, 7 October 2023 (UTC)

You have your work cut out for you as TFA is now featuring another good example: Affine symmetric group. ChatGPT struggles with the topic too. I reckon that it's possible to do better but it would be a thankless task currently. Andrew🐉(talk) 08:55, 8 October 2023 (UTC)

@Femke and Andrew Davidson: For what it's worth, you can ask ChatGPT or DeepL (which has a beta feature for this) to rewrite texts to make them easier to understand. Fed the Wikipedia lead section of Government shutdowns in the United States mentioned above and asked to improve its readability, ChatGPT produced the following:

In the U.S., government shutdowns happen when the funding needed to run the federal government isn't approved before the new fiscal year begins. During a shutdown, the government cuts back on agency activities and services, stops non-essential operations, puts non-essential workers on leave, and keeps only essential employees in departments that protect people or property. Shutdowns can also cause disruptions at the state, territorial, and local government levels.

Funding gaps started causing shutdowns in 1980, following a legal opinion from Attorney General Benjamin Civiletti. While this opinion wasn't consistently followed in the 1980s, since 1990, any funding gap lasting more than a few hours has led to a shutdown. As of September 2023, ten funding shutdowns have resulted in federal employees being placed on leave.

Major shutdowns include the 21-day shutdown in 1995–1996, during the Bill Clinton administration, due to disagreements over significant spending cuts; the 16-day shutdown in 2013, during the Barack Obama administration, caused by a dispute over implementing the Affordable Care Act (ACA); and the longest one, the 35-day shutdown in 2018–2019, during the Donald Trump administration, triggered by a disagreement over funding for expanding barriers on the U.S.–Mexico border.

Shutdowns disrupt government services and programs, leading to the closure of national parks and institutions. They also result in reduced government revenue as fees are lost, even though some furloughed employees eventually receive back pay. Furthermore, they negatively impact economic growth. For instance, during the 2013 shutdown, financial ratings agency Standard & Poor's reported on October 16 that the shutdown had already taken "$24 billion out of the economy" and "reduced annualized fourth-quarter 2013 GDP growth by at least 0.6 percent."

I wouldn't endorse every change, but would argue that the overall result is indeed improved readability. Andreas JN466 11:51, 8 October 2023 (UTC)

Get the latest headlines on your user page — just add {{Signpost-subscription}}.

Home

About

Archives

Newsroom

Subscribe

Suggestions