Documentation

h1. The OpenAIRE Research Graph

The OpenAIRE Research Graph is one of the largest open scholarly record collections worldwide, key in fostering Open Science and establishing its practices in the daily research activities. 

Conceived as a public and transparent good, populated out of data sources trusted by scientists, the Graph aims at bringing discovery, monitoring, and assessment of science back in the hands of the scientific community.

Imagine a vast collection of research products all linked together, contextualised and openly available. For the past ten years OpenAIRE has been working to gather this valuable record. It is a massive collection of metadata and links between scientific products such as articles, datasets, software, and other research products, entities like organisations, funders, funding streams, projects, communities, and data sources.

As of today, the OpenAIRE Research Graph aggregates around 450Mi metadata records with links collecting from 10K data sources trusted by scientists, including:

* Repositories registered in OpenDOAR or re3data.org

* Open Access journals registered in DOAJ

* Crossref

* Unpaywall

* ORCID

* Microsoft Academic Graph

* Datacite

After cleaning, deduplication, enrichment and full-text mining processes, the graph is analysed to produce statistics for OpenAIRE MONITOR (https://monitor.openaire.eu), the Open Science Observatory (https://osobservatory.openaire.eu), made discoverable via OpenAIRE EXPLORE (https://explore.openaire.eu) and programmatically accessible as described at https://develop.openaire.eu. 

Json dumps are also published on Zenodo.

TODO: image of high-level data model (entities and semantic relationships, we can draw here: https://docs.google.com/drawings/d/1c4s7Pk2r9NgV_KXkmX6mwCKBIQ-yK3_m6xsxB-3Km1s/edit)

h2. Graph Data Dumps

In order to facilitate users, different dumps are available. All are available under the "Zenodo community called OpenAIRE Research Graph":https://zenodo.org/communities/openaire-research-graph.

Here we provide detailed documentation about the full dump:

* Json dump: https://doi.org/10.5281/zenodo.3516917

* Json schema: https://doi.org/10.5281/zenodo.4238938 

[[Json schema]]

[[FAQ]]

h2. Graph provision processes

h3. OpenAIRE entity identifier & PID mapping policy

(copied from https://docs.google.com/document/d/1PnvZpmhbanJu3AeOT-zdIyMKIHoGKC4_Z0UtDFDZAeM/edit#)

OpenAIRE assign internal identifiers for each object it collects. 

By default, the internal identifier is generated as @sourcePrefix::md5(localId)@ where 

* @sourcePrefix@ is a namespace prefix of 12 chars assigned to the data source at registration time

* @localid@ is the identifier assigned to the object by the data source

After years of operation, we can say that:

* @localId@ are unstable

* objects can disappear from sources

* PIDs provided by sources that are not PID agencies (authoritative sources for a specific type of PID) are often wrong (e.g. pre-print with the DOI of the published version, DOIs with typos)

Therefore, when the record is collected from an authoritative source:

* the identity of the record is forged using the PID, like @pidTypePrefix::md5(lowercase(doi))@

* the PID is added in a @pid@ element of the data model.

When the record is collected from a source which is not authoritative for any type of PID:

* the identity of the record is forged as usual using the local identifier;

* the PID, if available, is added as @alternateIdentifier@s

As of November 2021, the following data sources are used as "PID authorities": 

| PID Type | Prefix (12 chars) | Authority |

| doi | @doi_________@ | Crossref, Datacite, Zenodo |

| pmc | @pmc_________@ | Europe PubMed Central, PubMed Central |

| pmid | @pmid________@ | Europe PubMed Central, PubMed Central |

| arXiv | @arXiv_______@ | arXiv.org e-Print Archive |

| handle | @handle______@ | any repository |

TODO: WHAT HAPPENS FOR RECORDS WITH BOTH pmc and pmid? pmc wins?

OpenAIRE also perform duplicate identification (see dedicated section for details).

All duplicates are "merged" together in a "representative record" which must be assigned to a dedicated OpenAIRE identifier (i.e. it cannot have the identifier of one of the aggregated record).

The following strategy is applied to generate the OpenAIRE identifier of a representative record, to ensure it is as stable as possible:

TODO

h3. Aggregation business logic by major sources

h4. DOIBoost: Crossref, Unpaywall, Microsoft Academic Graph, ORCID

The idea behind DOIBoost and its origin can be found in the paper (and related resources) at: 

* La Bruzzo S., Manghi P., Mannocci A. (2019) OpenAIRE's DOIBoost - Boosting CrossRef for Research. In: Manghi P., Candela L., Silvello G. (eds) Digital Libraries: Supporting Open Science. IRCDL 2019. Communications in Computer and Information Science, vol 988. Springer, doi:10.1007/978-3-030-11226-4_11 . Open Access version available at: https://doi.org/10.5281/zenodo.1441071

In short, the goal is to enrich the records available on Crossref with what's available on Unpaywall, Microsoft Academic Graph, ORCID intersecting all those datasets by DOI.

The generation of DOIBoost consists in the following phases:

1 Filter Crossref records that:

* have blank title

* have one of the following publishers: "Test accounts", "CrossRef Test Account"

* have no authors with valid names, where valid means: not blank and different from all strings in this list: @List(",", "none none", "none, none", "none &na;", "(:null)", "test test test", "test test", "test", "&na; &na;")@

* have "Addie Jackson" as author and "Elsevier BV" as publisher (empirically we say they are test records)

2 Intersect Crossref with Unpaywall by DOI (DOIBoost1). The records are enriched with 

* TODO: AUTHORS?

* one @instance@ with 

** the @best_oa_location@ of Unpaywall

** @color@ set as follows: @green@ if the host is a repository; @gold@ if the host is publisher and the journal is open access; @hybrid@ if the host is publisher, the journal is not open access but there is a license; @bronze@ if no license is available.

3 Intersect DOIBoost1 with ORCID (DOIBoost2). The records are enriched with the ORCID identifiers of their authors

4 Intersect DOIBoost2 with Microsoft Academic Graph (DOIBoost3). The records are enriched with:

* abstracts

* MAG identifiers of authors

* affiliation relationships

* subjects (MAG FieldsOfStudy)

* conference or journal information (in the @journal@ field) TODO: or @container@, in case of the dump?

* [TO BE REMOVED] instances with URL from MAG

5 Enrich DOIBoost3 with hosting data sources (@hostedby@) and access right information. In this phase we intersect DOIBoost3 with a dataset composed of journals from OpenAIRE, Crossref, and the ISSN gold list. Each journal comes with its International Standard Serial Numbers (issn, eissn, lissn) and, when available, a flag that tells if the journal is open access. The intersection is done on the basis of the International Standard Serial Numbers. The records with a @journal.[l|e]issn@ that match are enriched as follows:

* Each instance gain the `hostedby` information. 

* If the journal is open access, the access rights of the instances are also set to "Open Access" with "gold" route.

The hostedby of records that do not match are set to the "Unknown Repository".

h3. Deduplication business logic

h4. Deduplication business logic for research results 

Metadata records about the same scholarly work can be collected from different providers. Each metadata record can possibly carry different information because, for example, some providers are not aware of links to projects, keywords or other details. Another common case is when OpenAIRE collects one metadata record from a repository about a pre-print and another record from a journal about the published article. For the provision of statistics, OpenAIRE must identify those cases and “merge” the two metadata records, so that the scholarly work is counted only once in the statistics OpenAIRE produces. 

Duplicates among research results are identified among results of the same type (publications, datasets, software, other research products). If two duplicate results are aggregated one as a dataset and one as a software, for example, they will never be compared and they will never be identified as duplicates.

OpenAIRE supports different deduplication strategies based on the type of results.

*Methodology overview*

The deduplication process can be divided into two different phases: 

* Candidate identification (clustering)

* Decision tree

* Creation of representative record

The implementation of each phase is different based on the type of results that are being processed.

*Strategy for publications*

_Candidate identification (clustering)_

Due to the high number of metadata records collected by OpenAIRE, it would not be feasible to compute all possible comparisons between all metadata records.

The goal of this phase is to limit the number of comparisons by creating groups (or clusters) of records that are likely “similar”. Every record can be added to more than one group. 

The decision of inclusion in a group is performed by 2 clustering functions:

* Lowercase: doi (in pid list and alternate identifiers list)

* WordsStatsSuffixPrefixChain: suffixprefix with statistics on the full title (number_of_words & number_of_letters%10)

Example: 

If title is : “Search for the Standard Model Higgs Boson”

The clustering function produces 2 keys (i.e. adds the publication to two clusters): [5-3-seaardmod, 5-3-rchstadel]

_Desicision tree_

For each pair of publications in a cluster the following strategy (depicted in the figure below) is applied.

Cross comparison of the pid lists (in the @pid@ and @alternateid@ elements). If 50% common pids, levenshtein distance on titles with low threshold (0.9).

Otherwise, check if the number of authors and the title version is equal. If so, levenshtein distance on titles with higher threshold (0.99). 

The publications are matched as duplicate if the distance is higher than the threshold, in every other case they are considered as distinct publications.

!dedup-results.png!

_Creation of representative record_

TODO

*Strategy for datasets*

*Strategy for software*

*Strategy for other types of research products*

*Clustering functions*

_NgramPairs_

It produces a list of concatenations of a pair of ngrams generated from different words.

Example:

Input string: “Search for the Standard Model Higgs Boson”

Parameters: ngram length = 3

List of ngrams: “sea”, “sta”, “mod”, “hig”

Ngram pairs: “seasta”, “stamod”, “modhig”

_SuffixPrefix_

It produces ngrams pairs in a particular way: it concatenates the suffix of a string with the prefix of the next in the input string.

Example:

Input string: “Search for the Standard Model Higgs Boson”

Parameters: suffix and prefix length = 3

Output list: “ardmod” (suffix of the word “Standard” + prefix of the word “Model”), “rchsta” (suffix of the word “Search” + prefix of the word “Standard”)

h3. TODOs

* OpenAIRE entity identifier & PID mapping policy (started, to be completed by Claudio and/or Michele DB)

* Aggregation business logic by major sources:

** -Unpaywall integration-

** -Crossref integration- 

** -ORCID integration-

** -Cross cleaning actions: hostedBy patch-

** Scholexplorer business logic (relationship resolution)

** DataCite

** EuropePMC

** more….

* Deduplication business logic (started, to be completed by Michele DB)

** For research outputs ( -publications- , datasets, software, orp)

** For research organizations 

* Enrichment

** Mining business logic

** Deduction-based inference 

** Propagation business logic

* Post-cleaning business logic

* FAQ