OpenAIRE Research Graph » History » Revision 15
« Previous |
Revision 15/36
(diff)
| Next »
Alessia Bardi, 05/11/2021 03:53 PM
The OpenAIRE Research Graph¶
The OpenAIRE Research Graph is one of the largest open scholarly record collections worldwide, key in fostering Open Science and establishing its practices in the daily research activities.
Conceived as a public and transparent good, populated out of data sources trusted by scientists, the Graph aims at bringing discovery, monitoring, and assessment of science back in the hands of the scientific community.
Imagine a vast collection of research products all linked together, contextualised and openly available. For the past ten years OpenAIRE has been working to gather this valuable record. It is a massive collection of metadata and links between scientific products such as articles, datasets, software, and other research products, entities like organisations, funders, funding streams, projects, communities, and data sources.
As of today, the OpenAIRE Research Graph aggregates around 450Mi metadata records with links collecting from 10K data sources trusted by scientists, including:- Repositories registered in OpenDOAR or re3data.org
- Open Access journals registered in DOAJ
- Crossref
- Unpaywall
- ORCID
- Microsoft Academic Graph
- Datacite
After cleaning, deduplication, enrichment and full-text mining processes, the graph is analysed to produce statistics for OpenAIRE MONITOR (https://monitor.openaire.eu), the Open Science Observatory (https://osobservatory.openaire.eu), made discoverable via OpenAIRE EXPLORE (https://explore.openaire.eu) and programmatically accessible as described at https://develop.openaire.eu.
Json dumps are also published on Zenodo.
TODO: image of high-level data model (entities and semantic relationships, we can draw here: https://docs.google.com/drawings/d/1c4s7Pk2r9NgV_KXkmX6mwCKBIQ-yK3_m6xsxB-3Km1s/edit)
Graph Data Dumps¶
In order to facilitate users, different dumps are available. All are available under the Zenodo community called OpenAIRE Research Graph.
Here we provide detailed documentation about the full dump:
- Json dump: https://doi.org/10.5281/zenodo.3516917
- Json schema: https://doi.org/10.5281/zenodo.4238938
Graph provision processes¶
OpenAIRE entity identifier & PID mapping policy¶
(copied from https://docs.google.com/document/d/1PnvZpmhbanJu3AeOT-zdIyMKIHoGKC4_Z0UtDFDZAeM/edit#)
OpenAIRE assign internal identifiers for each object it collects.By default, the internal identifier is generated as
sourcePrefix::md5(localId)
where
sourcePrefix
is a namespace prefix of 12 chars assigned to the data source at registration timelocalid
is the identifier assigned to the object by the data source
localId
are unstable- objects can disappear from sources
- PIDs provided by sources that are not PID agencies (authoritative sources for a specific type of PID) are often wrong (e.g. pre-print with the DOI of the published version, DOIs with typos)
- the identity of the record is forged using the PID, like
pidTypePrefix::md5(lowercase(doi))
- the PID is added in a
pid
element of the data model.
- the identity of the record is forged as usual using the local identifier;
- the PID, if available, is added as @alternateIdentifier@s
As of November 2021, the following data sources are used as "PID authorities":
PID Type | Prefix (12 chars) | Authority |
doi | doi_________ |
Crossref, Datacite, Zenodo |
pmc | pmc_________ |
Europe PubMed Central, PubMed Central |
pmid | pmid________ |
Europe PubMed Central, PubMed Central |
arXiv | arXiv_______ |
arXiv.org e-Print Archive |
handle | handle______ |
any repository |
TODO: WHAT HAPPENS FOR RECORDS WITH BOTH pmc and pmid? pmc wins?
OpenAIRE also perform duplicate identification (see dedicated section for details).
All duplicates are "merged" together in a "representative record" which must be assigned to a dedicated OpenAIRE identifier (i.e. it cannot have the identifier of one of the aggregated record).
The following strategy is applied to generate the OpenAIRE identifier of a representative record, to ensure it is as stable as possible:
TODO
Aggregation business logic by major sources¶
DOIBoost: Crossref, Unpaywall, Microsoft Academic Graph, ORCID¶
The idea behind DOIBoost and its origin can be found in the paper (and related resources) at:
- La Bruzzo S., Manghi P., Mannocci A. (2019) OpenAIRE's DOIBoost - Boosting CrossRef for Research. In: Manghi P., Candela L., Silvello G. (eds) Digital Libraries: Supporting Open Science. IRCDL 2019. Communications in Computer and Information Science, vol 988. Springer, doi:10.1007/978-3-030-11226-4_11 . Open Access version available at: https://doi.org/10.5281/zenodo.1441071
In short, the goal is to enrich the records available on Crossref with what's available on Unpaywall, Microsoft Academic Graph, ORCID intersecting all those datasets by DOI.
The generation of DOIBoost consists in the following phases:
- have blank title
- have one of the following publishers: "Test accounts", "CrossRef Test Account"
- have no authors with valid names, where valid means: not blank and different from all strings in this list:
List(",", "none none", "none, none", "none &na;", "(:null)", "test test test", "test test", "test", "&na; &na;")
- have "Addie Jackson" as author and "Elsevier BV" as publisher (empirically we say they are test records)
- TODO: AUTHORS?
- one
instance
with- the
best_oa_location
of Unpaywall color
set as follows:green
if the host is a repository;gold
if the host is publisher and the journal is open access;hybrid
if the host is publisher, the journal is not open access but there is a license;bronze
if no license is available.
- the
3 Intersect DOIBoost1 with ORCID (DOIBoost2). The records are enriched with the ORCID identifiers of their authors
4 Intersect DOIBoost2 with Microsoft Academic Graph (DOIBoost3). The records are enriched with:- abstracts
- MAG identifiers of authors
- affiliation relationships
- subjects (MAG FieldsOfStudy)
- conference or journal information (in the
journal
field) TODO: orcontainer
, in case of the dump? - [TO BE REMOVED] instances with URL from MAG
hostedby
) and access right information. In this phase we intersect DOIBoost3 with a dataset composed of journals from OpenAIRE, Crossref, and the ISSN gold list. Each journal comes with its International Standard Serial Numbers (issn, eissn, lissn) and, when available, a flag that tells if the journal is open access. The intersection is done on the basis of the International Standard Serial Numbers. The records with a journal.[l|e]issn
that match are enriched as follows:
- Each instance gain the `hostedby` information.
- If the journal is open access, the access rights of the instances are also set to "Open Access" with "gold" route.
The hostedby of records that do not match are set to the "Unknown Repository".
Deduplication business logic¶
Deduplication business logic for research results¶
Metadata records about the same scholarly work can be collected from different providers. Each metadata record can possibly carry different information because, for example, some providers are not aware of links to projects, keywords or other details. Another common case is when OpenAIRE collects one metadata record from a repository about a pre-print and another record from a journal about the published article. For the provision of statistics, OpenAIRE must identify those cases and “merge” the two metadata records, so that the scholarly work is counted only once in the statistics OpenAIRE produces.
Duplicates among research results are identified among results of the same type (publications, datasets, software, other research products). If two duplicate results are aggregated one as a dataset and one as a software, for example, they will never be compared and they will never be identified as duplicates.
OpenAIRE supports different deduplication strategies based on the type of results.
Methodology overview
The deduplication process can be divided into two different phases:- Candidate identification (clustering)
- Decision tree
- Creation of representative record
The implementation of each phase is different based on the type of results that are being processed.
Strategy for publications
Candidate identification (clustering)
Due to the high number of metadata records collected by OpenAIRE, it would not be feasible to compute all possible comparisons between all metadata records.The goal of this phase is to limit the number of comparisons by creating groups (or clusters) of records that are likely “similar”. Every record can be added to more than one group.
The decision of inclusion in a group is performed by 2 clustering functions:
- Lowercase: doi (in pid list and alternate identifiers list)
- WordsStatsSuffixPrefixChain: suffixprefix with statistics on the full title (number_of_words & number_of_letters%10)
Example:
If title is : “Search for the Standard Model Higgs Boson”
The clustering function produces 2 keys (i.e. adds the publication to two clusters): [5-3-seaardmod, 5-3-rchstadel]
Desicision tree
For each pair of publications in a cluster the following strategy (depicted in the figure below) is applied.
Cross comparison of the pid lists (in the pid
and alternateid
elements). If 50% common pids, levenshtein distance on titles with low threshold (0.9).
Otherwise, check if the number of authors and the title version is equal. If so, levenshtein distance on titles with higher threshold (0.99).
The publications are matched as duplicate if the distance is higher than the threshold, in every other case they are considered as distinct publications.
Creation of representative record
TODO
Strategy for datasets
Strategy for software
Strategy for other types of research products
Clustering functions
NgramPairs
It produces a list of concatenations of a pair of ngrams generated from different words.
Example:
Input string: “Search for the Standard Model Higgs Boson”
Parameters: ngram length = 3
List of ngrams: “sea”, “sta”, “mod”, “hig”
Ngram pairs: “seasta”, “stamod”, “modhig”
SuffixPrefix
It produces ngrams pairs in a particular way: it concatenates the suffix of a string with the prefix of the next in the input string.
Example:
Input string: “Search for the Standard Model Higgs Boson”
Parameters: suffix and prefix length = 3
Output list: “ardmod” (suffix of the word “Standard” + prefix of the word “Model”), “rchsta” (suffix of the word “Search” + prefix of the word “Standard”)
TODOs¶
- OpenAIRE entity identifier & PID mapping policy (started, to be completed by Claudio and/or Michele DB)
- Aggregation business logic by major sources:
Unpaywall integrationCrossref integrationORCID integrationCross cleaning actions: hostedBy patch- Scholexplorer business logic (relationship resolution)
- DataCite
- EuropePMC
- more….
- Deduplication business logic (started, to be completed by Michele DB)
- For research outputs (
publications, datasets, software, orp) - For research organizations
- For research outputs (
- Enrichment
- Mining business logic
- Deduction-based inference
- Propagation business logic
- Post-cleaning business logic
- FAQ
Updated by Alessia Bardi about 3 years ago · 15 revisions