View source for Wikidata/Notes/Inclusion syntax v0.2

{{Message | type = important | Obsolete | This draft is obsolete and only kept for reference. Please refer to ''[[Wikidata/Notes/Inclusion_syntax]]'' for the current version }}
{{Message | type = attention | '''New Version (January 2013)'''
This is a completely reworked proposal with little resemblance to the [[Wikidata/Notes/Inclusion_syntax_v0.1|previous draft]]!
}}
This page describes the data inclusion syntax for the Wikibase client, by which the properties of data items can be included and rendered on a wiki page using templates.

:Note: Please read ''[[Wikidata/Data model]]'' before reading this document. Understanding the underlying data model is very useful when discussing the way the data can be accessed and rendered on wiki pages.

{{User:Markus Krötzsch/EN|Name is misleading, this is a functional spec for client side data access and formatting. It's not about syntax}}
{{User:Markus Krötzsch/EN|This should be turned into an object-oriented functional spec, with the syntax factored out into a parser function binding, Lua binding, etc}}

== Accessing Item Data ==

Properties of a Wikidata Item can be used via the #property parser function:

  <nowiki>{{#property:population}}</nowiki>

This will provide the value of the <code>population</code> property of the page's default item. The default item is the Wikidata item
that is associated with this page by virtue of its <code>wikilink</code> property.

To access properties of a different item, the <code>item</code> has to be specified explicitly, either by id or using the associated Wiki page:

  <nowiki>{{#property:population|item=id/q12345}}</nowiki>

or 

  <nowiki>{{#property:population|item=en/Germany}}</nowiki>

Item data is cached in memory, so accessing several properties of the same item is efficient.

{{User:Markus Krötzsch/EN|The exact form of the different types of item ids is not yet decided. However, the form used for the item parameter here should be the same as the form used in (inter)wiki links when linking to items (resp. item pages) on Wikidata. For title-based links, we could use colon-syntax: "en:Germany", or just "Germany" to use the local site's language; For IDs, we could use "/q1234" or "qid/q1234" or "::q1234" or whatever. }}
{{User:Markus Krötzsch/EN|How to access values from complex data types, like geo-coordinates? Use parts? or use dots, "coord.lat"? or slashes, "coord/lat"?}}

== Value Formatting ==

The item value is returned as wikitext per default, with some formatting applied where appropriate. For instance,
dates and numbers are formatted using the formatting rules of the page's content language, item references are
turned into local wiki links, etc. The concrete formatting rules depend on the property's data type.

To access the raw value, e.g. to apply custom formatting, use the <code>format</code> option:

  <nowiki>{{#property:population|format=raw}}</nowiki>

This would return something like ''1234567'' instead of using e.g. the en-US formatted version ''1,234,567''.

Unit conversion can also be done on the fly:

  <nowiki>{{#property:area|unit=miles^2|precision=0.1}}</nowiki>

This would return the area in square miles, to one digit after the decimal point.

The following parameters can be used to control the function's output:

;precision: the precision for the output of scalar values, such as "1km" or "1 ounce". If not given, this may be derived based on the unit used for display, and on the value's precision qualifier.
;unit: the unit of measurement for display (implies a precision of 1/10th (TBD?!) of that unit, if the precision isn't given implicitly).
;language: the language to prefer for the value, as a fall-back list of language codes. Per default, the page's content language is used.
;source: select a specific statement to use by providing a source reference identifier (see [[#Explicit Selection of Statements]]).
;part: which part of the property's statement(s) to show (see [[#Statement Parts]]). By default, the value part is shown (if one exists for the respective data type).

== Statement Parts ==

In Wikidata, properties don't simply have one value. Instead, their have ''statements'' consisting
of several ''parts'' (roughly equivalent to the "snaks" in the data model), one of it is (usually) "the" value. 
Some types of properties may not have a single value - e.g.
a geo-location would have a latitude and longitude part, but no "value".

All parts of the statement can be accessed using the #property function:

  <nowiki>{{#property:population|part=source}}</nowiki>
  <nowiki>{{#property:population|part=accuracy}}</nowiki>
  <nowiki>{{#property:population|part=timestamp|precision=year}}</nowiki>

These would provide the source(s), the accuracy, and the timestamp of the statement about the population. 
The timestamp would be given as a year, even if provided in more detail. The source references are themselves
complex objects that require templates for rendering (see below).
 
If the part is not given explicitly, the default (value) part of the statement is used (as in the examples in the previous section). 
If the statement has no default value part (e.g. for geo-coordinates), and no part is explicitly specified, a warning message is returned
instead of any property value.

The following parts are well known and/or have a special meaning:

;value: the "main" value part (if one exists - some data types like geo-coordinates don't have "a" value).
;source: any source references 
;accuracy: the margin of error (for scalar values)
;timestamp: the point in the the value applies to
;preference: preference value(s) for the present statement(s), such as "preferred", "other", "unsourced" or "deprecated".

{{User:Markus Krötzsch/EN|(most) qualifiers are user defined. Qualifiers have data types.}}

Part names given here are essentially the names of qualifiers (that is: of snak types), in the data model. Additional qualifiers may be added and used 
on the repository, though.

There are also some "virtual parts" that have no correspondence in the data model but may be useful when showing properties:

;details-link: link/button to expand the display of the property (e.g. to show a pop-up with all values/statements). (TBD: how to define/customize the appearance?)
;edit-link: link/button to trigger an edit interface for the given property (should show all values!)
;warnings: any warnings associated with this value, using the appropriate icons and links. This is an automatic ("virtual") part, not based on an actual qualifier/snak in the data. Typical warnings are
:* ''disputed'' if there are multiple default values, but the property is not marked as inherently multi-value. 
:* ''unsourced'' if an unsourced value is included, but the property is not marked as inherently unsourced. 
:* ''deprecated'' if a deprecated value is included
:* ''unreviewed'' (for future use with flagged revisions) the value was not yet reviewed by an established editor.

In addition to the above, some parts are taken from the property declaration, not the item data itself:

;label: the property's label (in the page's content language)
;type: the property's data type identifier.

== Coalescing Values ==
{{User:Markus Krötzsch/EN|"Aggregating" may be a better term than "coalescing"}}
{{User:Markus Krötzsch/EN|Perhaps "list of values" should be the only type of aggregation.}}
{{User:Markus Krötzsch/EN|Perhaps parts should not be aggregated individually?}}

Properties don't necessarily have a single value. They may have multiple values  (more precisely, multiple statements), 
either because they are inherently multivalued, or because conflicting opinions exist about the value, or because the
value changes over time.

Each statement has a "preference" value, which may be "preferred", "other", "unsourced" and "deprecated". When determining the statement to use for a call to the #property function, "preferred" statements are picked over "other" statements, "other" statements are preferred over "unsourced" onces, etc. 

However, there may still be multiple statements that are equally "strong": There may be multiple "preferred" values, or multiple "other" values (and no preferred values), etc.

In that case, these statements are combined or "coalesced" to form a single virtual statement: all parts of the statements are combined in a way appropriate for they respective data types. The list of sources for the statements are concatenated, the worst accuracy is chosen, the values are combined into ranges or lists, depending on their type (note that while there may be several statements with different values for a given property, they all have the same data type, namely, the type specified in the property declaration). For example, if there are three values for the population given by different sources, and there's no agreement on which source should be authoritative, this would be represented by three statements for a single property:

{| class="wikitable" |
! property !! value !! accuracy !! timestamp !! source !! preference
|-
| population || 263455 || +/-200 || 2010 || Foo || preferred
|-
| population || 251104 || +/-100 || 2011 || Bar || preferred
|-
| population || 268122 || +/-1% || January 2010 || Quux || preferred
|}

There may be more non-preferred (e.g. older) values:

{| class="wikitable" |
! property !! value !! accuracy !! timestamp !! source !! preference
|-
| population || 261108 || +/-200 || 2009 || Foo || other
|-
| population || 250104 || +/-80 || 2009 || Acme || other
|}

In this case, the default values would automatically be coalesced into a range:

   <nowiki>{{#property:population}}</nowiki>

Would evaluate to something like this (depending on the property definition):

:''251,104 &ndash; 268,122''

Other parts are also coalesced, essentially forming a single statement:

{| class="wikitable" |
! property !! value !! accuracy !! timestamp !! source !! preference
|-
| population || 251104 - 268122 || +/-1% || 2010 - 2011 || Foo, Bar, Quux || preferred
|}

The way different parts are combined depends on their type and semantics: scalar values are combined into ranges,
texts, sources and item references are combined into lists, the accuracy is using the maximum (worst accuracy),
the timestamp uses a range cut be the minimum accuracy (in this case, the "January" part is dropped because other
timestamps didn't provide a month), etc.

{{User:Markus Krötzsch/EN|More details about aggregation for different data types (and qualifier types).}}

=== Explicit Selection of Statements ===

To avoid automatic coalescing even if there are several equally strong statements, the desired statement can
be selected explicitly, using it's source as an identifier (Note: this assumes that a single sources does not
make multiple contradicting claims &ndash; if it does, the source should be made more precise, e.g. by giving the page
and paragraph). 

So, in the above example, this could be used:

   <nowiki>{{#property:population|source=Bar}}</nowiki>

This would pick the value given by source "Bar":

:''251,104''

{{User:Markus Krötzsch/EN|TBD: still unclear how source/reference identifier are maintained.}}

== Multiple Statements ==
If a property has multiple statements, it is sometimes desirable to simply list all or some of them in detail.
With the help of a scripting language like Lua, this is easy enough using a for-loop. However, we need
some help to achieve the same using traditional MediaWiki template syntax (i.e. parser functions).

To this end, the <nowiki>{{#property-values}}</nowiki> can be used:

    <nowiki>{{#property-values:area|template=country-area-info|unit=miles^2|precision=0.1}}</nowiki>

This would call the template ''country-area-info'' for each statement of the property ''area''. Inside the the template, the respective
property (e.g. ''area'') would have only a single value. Other parameters (e.g. ''unit'') are passed to the template as simple
parameters.

{{User:Markus Krötzsch/EN|This means template output depends on context beyond template parameters, which is bound to screw up caching.}}

If ''Template:Country-area-info'' was supposed to generate a single table row, its implementation could look something like this:

<pre>
     |-
     | {{#property:area|unit={{{unit|km^2}}}|precision={{{precision|1}}}}} {{#property:area|part=accuracy}} {{{unit}}}
     | {{#property:area|part=timestamp|precision=year}} <ref>{{#property:area|part=source}}</ref>
</pre>

No coalescing would take place here, since the additional statements for the property would be masked by the <nowiki>{{#property-values}}</nowiki> function.

So, using this mechanism, a table if different opinions about the area could be constructed like this:

<pre>
{| class="wikitable" |
! value !! accuracy !! timestamp !! source
{{#property-values:area|template=country-area-info|unit=miles^2|precision=0.1}}
|}
</pre>

which would result in something like:

{| class="wikitable" |
! Area !! Time
|-
| 23,455 +/-20 miles^2 || 2011<sup>[Foo]</sup>
|-
| 25,104 +/-15 miles^2 || 2010<sup>[Bar]</sup>
|-
| 26,822 +/-2% miles^2 || May 2010<sup>[Quux]</sup>
|}

{{User:Markus Krötzsch/EN|the above could of course be formatted more nicely, the example is deliberately kept simple.}}
{{User:Markus Krötzsch/EN|Pending: options for controlling the maximum number of items, minimum required strength, etc.}}

== Changing the Default Item ==
{{User:Markus Krötzsch/EN|This feature is syntactic sugar and should be marked as optional}}

If a page or template wants to make a different item the default item, this can be done using the <nowiki>{{#data-item}}</nowiki> function. 
For instance, on the page ''Germany'' on the English language Wikipedia, the default item would (per definition) be ''en/Germany''. So

  <nowiki>{{#property:population}}</nowiki>

would be shorthand for 

  <nowiki>{{#property:population|item=en/Germany}}</nowiki>

To override this, the following syntax can then be used:

  <nowiki>{{#data-item:id/q54321}}</nowiki>

Now,

  <nowiki>{{#property:population}}</nowiki>

would be shorthand for 

  <nowiki>{{#property:population|item=id/q54321}}</nowiki>  

This may be of limited used directly inside an article page (the need to do this would indicate that there's something wrong with the
language links or the article's scope). But it is expected that this mechanism is quite useful inside templates. Maybe there's an extra
article (and data item) about the Germany Economy, with some overview data like GNP, etc. It may then be useful to show an infobox
about the economy directly on the page about Germany itself. With the help of <nowiki>{{#data-item}}</nowiki>, we could do the following
on the ''Germany'' page:

  <nowiki>{{national economy box|item=en/Economy of Germany}}</nowiki>

Inside ''Template:National_economy_box'', the ''<nowiki>{{{item}}}</nowiki>'' parameter could now be used directly to fetch data properties
from the desired item:

  <nowiki>{{#property:GNP|item={{{item}}}}}</nowiki>

But this is cumbersome (<nowiki>item={{{item}}}</nowiki> all over the place) and also error prone (missing <nowiki>item={{{item}}}</nowiki>
in some places). It's nicer to just set the default item for the template:

  <nowiki>{{#data-item:{{{item}}}}}</nowiki>

This sets the data item for the present scope (implemented using a preprocessor frame), so the item doesn't have to be given explicitly:

   <nowiki>{{#property:GNP}}</nowiki>

Note that <nowiki>{{#data-item}}</nowiki> sets the default item for the present ''scope'' (e.g. the template), not the entire page
(which would be confusing)! The present "scope" is basically the wikitext stored on a single wiki page - "calling" a template creates a new scope. 

{{User:Markus Krötzsch/EN|If "deeper" template calls should inherit the changed default item from "further up", this would mess with chunk caching in Parsoid, since it depends on context, no on the parameters.}}
{{User:Markus Krötzsch/EN|Parsoid evaluates parser functions in parallel, but #data-item would have to be evaluated before all calls to #property, otherwise the results become random}}

== Special properties ==

'''TBD:''' Describe special properties like ''edit-link'', ''id'', ''alias'', etc.

== Formatting Sources ==

'''TBD:''' mechanisms for formatting sources and source references in compliance with the current citation system and existing citation templates.

== Property Definitions ==

'''TBD:''' Describe what properties properties have, how they are maintained, etc

----
Item properties reference the Property declaration. Property declarations are heavily cached on repository and client. They contain:

* Data type (also a reference, also heavily cached)
* Labels in several languages
* "no source needed" flag (don't assign "unsourced" status, even if no source is given)
* "multi-value" flag (multiple default values don't constitute a conflict/dispute)

[[Category:Wikidata]]