Wikipedia:Wikipedia Signpost/2023-04-03/Recent research

Recent research

Language bias: Wikipedia captures at least the "silhouette of the elephant", unlike ChatGPT

By Tilman Bayer

A monthly overview of recent academic research about Wikipedia and other Wikimedia projects, also published as the Wikimedia Research Newsletter.

"A Perspectival Mirror of the Elephant: Investigating Language Bias on Google, ChatGPT, Wikipedia, and YouTube"

The blind men and the elephant
(wall relief in Northeast Thailand)

This arXiv preprint (which according to the authors grew out of a student project for a course "Critical Thinking in Data Science" at Harvard University) finds that

[...] Google and its most prominent returned results -- Wikipedia and YouTube, simply reflect the narrow set of cultural stereotypes tied to the search language for complex topics like "Buddhism," "Liberalism," "colonization," "Iran" and "America." Simply stated, they present, to varying degrees, distinct information across the same search in different languages (we call it 'language bias'). Instead of presenting a global picture of a complex topic, our online searches turn us into the proverbial blind person touching a small portion of an elephant, ignorant of the existence of other cultural perspectives.

Regarding Wikipedia, the authors note it "is an encyclopedia that provides summaries of knowledge and is written from a neutral point of view", concluding that

[...] even though the tones of voice and views do not differ much in Wikipedia articles across languages, topic coverage in Wikipedia articles tends to be directed by the dominant intellectual traditions and camps across different language communities, i.e., a French Wikipedia article focuses on French thinkers, and a Chinese article stresses on Chinese intellectual movements. Wikipedia’s fundamental principles or objectives filter language bias, making it heavily rely on intellectual and academic traditions.

While the authors employ some quantitative methods to study the bias on the other three sites (particularly Google), the Wikipedia part of the paper is almost entirely qualitative in nature. It focuses on an in-depth comparison of a small set of (quite apparently non-randomly chosen) article topics across languages, not unlike various earlier studies of language bias on Wikipedia (e.g. on the coverage of the holocaust in different languages, see our previous coverage here and here). Unfortunately, despite asserting that "there has been a lack of investigation into language bias on platforms such as Google, ChatGPT, Wikipedia, and YouTube", the paper fails to cite such such earlier research (which has also included quantitative results, such as those represented in the "Wikipedia Diversity Observatory", which among other things includes data on topic coverage across 300+ Wikipedia languages).

The first and largest part of the paper's examination of Wikipedia's coverage concerns articles about buddhism and various subtopics, in the English, French, German, Vietnamese, Chinese, Thai and Nepali Wikipedias. The authors indicate that they chose this topic starting out from the observation that

To Westerners, Buddhism is generally associated with spirituality, meditation, and philosophy, but people who primarily come from a Vietnamese background might see Buddhism as closely tied to the lunar calendar, holidays, mother god worship, and capable of bringing good luck. One from a Thai culture might regard Buddhism as a canopy against demons, while a Nepali might see Buddhism as a protector to destroy bad karma and defilements.

Somewhat in confirmation of this hypothesis, they find that

Compared to Google’s language bias, we find that Wikipedia articles’ content titles mainly differ in topic coverage but not much in tones of voice. The preferences of topics tend to correlate with the dominant intellectual traditions and camps in different language communities.

However, the authors also observe that "randomness is involved to some degree in terms of topic coverage on Wikipedia", defying overly simplistic bias expectations based on intellectual traditions. E.g.

Looking at the Chinese article on “Buddhism,” it addresses topics like “dharma name,” “cloth,” and “hairstyle” that do not exist on other languages’ pages. There are several potential causes for its special treatment on these issues. First, many Buddhist texts, such as the Lankavatara Sutra (楞伽经) and Vinaya Piṭaka (律藏), that address these issues were translated into Chinese during medieval China, and these texts are still widely circulated in China today. Second, according to the official central Chinese government statistics, there are over 33,000 monasteries in China, so people who are interested in writing Wikipedia articles might think it is helpful to address these issues on Wikipedia. However, like the pattern in the French article, Vietnam, Thailand, and Nepal all have millions of Buddhist practitioners, and the Lankavatara Sutra and Vinaya Piṭaka are also widely circulated among South Asian Buddhist traditions, but their Wikipedia pages do not address these issues like the Chinese article.

A second, shorter section focuses on comparing Wikipedia articles on liberalism and marxism across languages. Among other things, it observes that the "English article has a long section on Keynesian economics", likely due to its prominent role in the New Deal reforms in the US in the 1930. In contrast,

In the French article on liberalism, the focus is not solely on the modern interpretation of the term but rather on its historical roots and development. It traces its origins from antiquity to the Renaissance period, with a focus on French history. It also highlights the works of French theorists such as Montesquieu and Tocqueville [...]. The Italian article has a lengthy section on “Liberalism and Christianity” because liberalism can be seen as a threat to the catholic church. Hebrew has a section discussing the Zionist movement in Israel. The German article is much shorter than the French, Italian, and Hebrew ones. Due to Germany’s loss in WWII, its post-WWII state was a liberal state and was occupied by the Allied forces consisting of troops from the U.S., U.K., France, and the Soviet Union. This might have influenced Germany’s perception and approach to liberalism.

Among other proposals for reducing language bias on the four sites, the paper proposes that

"[Wikipedia] could potentially invite scholars to contribute articles in other languages to improve the multilingual coverage of the site. Additionally, Wikipedia could merge non-overlapping sections of articles on the same term but written in different languages into a single article, like how multiple branches of code are merged on GitHub. Like Google, Wikipedia could translate the newly inserted paragraphs into the user’s target language and place a tag to indicate its source language.

Returning to their title metaphor, the authors give Wikipedia credit for at least "show[ing] a rough silhouette of the elephant", whereas e.g. Google only "presents a piece of the elephant based on a user’s query language". However, "the silhouette – topic coverage – differs by language. [Wikipedia] writes in a descriptive tone and contextualizes first-person narratives and subjective opinions as cultural, historical, or religious phenomena." YouTube, on the other hand, "displays the ‘color’ and ‘texture’ of the elephant as it incorporates images and sounds that are effective in invoking emotions", but its top-rated videos "tend to create a more profound ethnocentric experience as they zoom into a highly confined range of topics or views that conform to the majority’s interests".

The papers singles out the new AI-based tools as particularly problematic regarding language bias:

The problem with language bias is compounded by ChatGPT. As it is primarily trained on English language data, it presents the Anglo-American perspective as truth [even when giving answers in other languages] – as if it were the only valid knowledge.

On the other hand, the paper's examination of the biases of "ChatGPT-Bing" [sic] highlights among other concerns its reliance on Wikipedia:

[...] all responses list Wikipedia articles as its #1 source, which means that language bias in Wikipedia articles is inevitably permeated in ChatGPT-Bing’s answers.

Briefly

• See the page of the monthly Wikimedia Research Showcase for videos and slides of past presentations.

Other recent publications

Other recent publications that could not be covered in time for this issue include the items listed below. Contributions, whether reviewing or summarizing newly published research, are always welcome.

"A systematic review of Wikidata in Digital Humanities projects"

From the abstract:

"A systematic review was conducted to identify and evaluate how DH [Digital Humanities] projects perceive and utilize Wikidata, as well as its potential and challenges as demonstrated through use. This research concludes that: (1) Wikidata is understood in the DH projects as a content provider, a platform, and a technology stack; (2) it is commonly implemented for annotation and enrichment, metadata curation, knowledge modelling, and Named Entity Recognition (NER); (3) Most projects tend to consume data from Wikidata, whereas there is more potential to utilize it as a platform and a technology stack to publish data on Wikidata or to create an ecosystem of data exchange; and (4) Projects face two types of challenges: technical issues in the implementations and concerns with Wikidata’s data quality."

"Leveraging Wikipedia article evolution for promotional tone detection"

From the abstract: From the abstract:

"In this work we introduce WikiEvolve, a dataset for document-level promotional tone detection. Unlike previously proposed datasets, WikiEvolve contains seven versions of the same article from Wikipedia, from different points in its revision history; one with promotional tone, and six without it. This allows for obtaining more precise training signal for learning models from promotional tone detection. [...] In our experiments, our proposed adaptation of gradient reversal improves the accuracy of four different architectures on both in-domain and out-of-domain evaluation."

"Mapping Process for the Task: Wikidata Statements to Text as Wikipedia Sentences"

From the abstract:

"The shortage of volunteers brings to Wikipedia many issues, including developing content for over 300 languages at the present. Therefore, the benefit that machines can automatically generate content to reduce human efforts on Wikipedia language projects could be considerable. In this paper, we propose our mapping process for the task of converting Wikidata statements to natural language text (WS2T) for Wikipedia projects at the sentence level. The main step is to organize statements, represented as a group of quadruples and triples, and then to map them to corresponding sentences in English Wikipedia. We evaluate the output corpus in various aspects: sentence structure analysis, noise filtering, and relationships between sentence components based on word embedding models."

Among other examples given in the paper, a Wikidata statement involving the items and properties Q217760, P54, Q221525 and P580 is mapped to the Wikipedia sentence "On 30 January 2010, Wiltord signed with Metz until the end of the season."

Judging from the paper's citations, the authors appear to have been unaware of the Abstract Wikipedia project, which is pursuing a closely related effort.

References

This page is a draft for the next issue of the Signpost. Below is some helpful code that will help you write and format a Signpost draft. If it's blank, you can fill out a template by copy-pasting this in and pressing 'publish changes': {{subst:Wikipedia:Wikipedia Signpost/Templates/Story-preload}}

Images and Galleries

Sidebar images To put an image in your article, use the following template (link): [[File:|center|300px|alt=TKTK]] O frabjous day. {{Wikipedia:Wikipedia Signpost/Templates/Filler image-v2 |image = |size = 300px |alt = TKTK |caption = |fullwidth = no }} This will create the file on the right. Keep the 300px in most cases. If writing a 'full width' article, change |fullwidth=no to |fullwidth=yes. Inline images Placing {{Wikipedia:Wikipedia Signpost/Templates/Inline image |image = |size = 300px |align = center |alt = Placeholder alt text |caption = CAPTION }} (link) will instead create an inline image like below [[File:|300px|center|alt=Placeholder alt text]] CAPTION Galleries To create a gallery, use the following <gallery mode = packed | heights = 200px> |Caption for second image </gallery> to create

Quotes

Framed quotes “ Lorem ipsum dolor sit amet... ” — AUTHOR, SOURCE To insert a framed quote like the one on the right, use this template (link): {{Wikipedia:Wikipedia Signpost/Templates/Filler quote-v2 |1 = |author = |source = |fullwidth = }} If writing a 'full width' article, change |fullwidth=no to |fullwidth=yes. Pull quotes To insert a pull quote like “ Lorem ipsum dolor sit amet... ” use this template (link): {{Wikipedia:Wikipedia Signpost/Templates/Quote |1 = |source = }} Long quotes To insert a long inline quote like The goose is on the loose! The geese are on the lease! — User:Oscar Wilde — Quotations Notes from the Underpoop use this template (link): {{Wikipedia:Wikipedia Signpost/Templates/block quote | text = | by = | source = | ts = | oldid = }}

Side frames

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. A caption Side frames help put content in sidebar vignettes. For instance, this one (link): {{Wikipedia:Wikipedia Signpost/Templates/Filler frame-v2 |1 = Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. |caption = A caption |fullwidth = no }} gives the frame on the right. This is useful when you want to insert non-standard images, quotes, graphs, and the like. Example − Graph/Charts Further information: Category:Graph, chart and plot templates A caption For example, to insert the {{Graph:Chart}} generated by {{Graph:Chart |width=250|height=100|type=line |x=1,2,3,4,5,6,7,8|y=10,12,6,14,2,10,7,9 }} in a frame, simple put the graph code in |1= {{Wikipedia:Wikipedia Signpost/Templates/Filler frame-v2 |1= {{Graph:Chart |width=250|height=100|type=line |x=1,2,3,4,5,6,7,8|y=10,12,6,14,2,10,7,9 }} |caption=A caption |fullwidth=no }} to get the framed Graph:Chart on the right. If writing a 'full width' article, change |fullwidth=no to |fullwidth=yes.

Two-column vs full width styles

If you keep the 'normal' preloaded draft and work from there, you will be using the two-column style. This is perfectly fine in most cases and you don't need to do anything. However, every time you have a |fullwidth=no and change it to |fullwidth=yes (or vice-versa), the article will take that style from that point onwards (|fullwidth=yes → full width, |fullwidth=no → two-column). By default, omitting |fullwidth= is the same as putting |fullwidth=no and the article will have two columns after that. Again, this is perfectly fine in most cases, and you don't need to do anything. However, you can also fine-tune which style is used at which point in an article. To switch from two-column → full width style midway in an article, insert {{Wikipedia:Wikipedia Signpost/Templates/Signpost-block-end-v2}} {{Wikipedia:Wikipedia Signpost/Templates/Signpost-block-start-v2|fullwidth=yes}} where you want the switch to happen. To switch from full width → two-column style midway in an article, insert {{Wikipedia:Wikipedia Signpost/Templates/Signpost-block-end-v2}} {{Wikipedia:Wikipedia Signpost/Templates/Signpost-block-start-v2|fullwidth=no}} where you want the switch to happen.

Article series

To add a series of 'related articles' your article, use the following code Related articles Visual Editor Five, ten, and fifteen years ago 1 January 2023 VisualEditor, endowment, science, and news in brief 5 August 2015 HTTPS-only rollout completed, proposal to enable VisualEditor for new accounts 17 June 2015 VisualEditor and MediaWiki updates 29 April 2015 Security issue fixed; VisualEditor changes 4 February 2015 More articles Draft namespace, VisualEditor meetings 25 December 2013 Sockpuppet investigations, VisualEditor, Wikidata's birthday, and more 6 November 2013 Last call for Wiki Loves Monuments; Community–WMF tension over VisualEditor 25 September 2013 Making Wikipedia more accessible 4 September 2013 Gallery improvements launch on Wikipedia 28 August 2013 The VisualEditor Beta and the path to change 31 July 2013 Defining consensus; VisualEditor default state; expert and layperson terms in article titles 31 July 2013 VisualEditor in midst of game-changing deployment series 3 July 2013 VisualEditor will "change world history" 12 June 2013 VisualEditor, elections, bots, and more 12 June 2013 The Visual Editor: Where are we now, and where are we headed? 25 March 2013 Visual Editor "on schedule" for July rollout 18 March 2013 Looking ahead to 2013 7 January 2013 The new Visual Editor gets a bit more visual 10 December 2012 Second Visual Editor prototype launches 25 June 2012 Visual editor demo launched, hailed as "most important change to our user experience ... ever"; but elsewhere over-hasty deployments criticised 19 December 2011 Full steam ahead on Visual Editor, the avoidance of lock-breaking and 1.18b1 release 7 November 2011 {{Signpost series |type = sidebar-v2 |tag = VisualEditor |seriestitle = Visual Editor |fullwidth = no }} or {{Signpost series |type = sidebar-v2 |tag = VisualEditor |seriestitle = Visual Editor |fullwidth = yes }} will create the sidebar on the right. If writing a 'full width' article, change |fullwidth=no to |fullwidth=yes. A partial list of valid |tag= parameters can be found at here and will decide the list of articles presented. |seriestitle= is the title that will appear below 'Related articles' in the box. Alternatively, you can use {{Signpost series |type = inline |tag = VisualEditor |tag_name = visual editor |tag_pretext = the }} at the end of an article to create For more Signpost coverage on the visual editor see our visual editor series. If you think a topic would make a good series, but you don't see a tag for it, or that all the articles in a series seem 'old', ask for help at the WT:NEWSROOM. Many more tags exist, but they haven't been documented yet.

Links and such

This department (Recent research): Next issue · Next next issue · Search for previous Arbitration report: Arbitration · Arbcom · Archives · Cases · Enforcements Discussion report: News and notes: In the media: Google News · TechMEME By the way, the template that you're reading right now is {{Editnotices/Group/Wikipedia:Wikipedia Signpost/Next issue}} (edit). A list of the preload templates for Signpost articles can be found here.

← Previous "Recent research"

In this issue

3 April 2023

From the editor

News and notes

In the media

Arbitration report

Featured content

Recent research

From the archives

Disinformation report

+ Add a comment

Discuss this story

These comments are automatically transcluded from this article's talk page. To follow comments, add the page to your watchlist. If your comment has not appeared here, you can try .

• I wonder which Wikipedia has the most cosmopolitan POV. If such a thing is even quantifiable in the first place... even the papers cited here just took individual articles into account. W. Tell DCCXLVI (^{talk to me!}/_c) 17:40, 3 April 2023 (UTC)

  A metric could be, for each Wikipedia: # of references used in other languages/total # of references used... in the 1000 articles every Wikipedia should have. The higher the ratio, the more cosmopilitan? Wikipedias in dutch or german might be more diverse as their users in general can read more languages. Cbrescia (talk) 06:39, 11 April 2023 (UTC)

Draft or not

Is this a draft paper or a published one? PAC2 (talk) 21:16, 3 April 2023 (UTC)

The "elephant" paper? It's a preprint, as mentioned in the opening sentence of the review. Regards, HaeB (talk) 18:10, 9 April 2023 (UTC)

The Signpost is written by editors like you — join in!

Home

About

Archives

Newsroom

Subscribe

Suggestions