Project

General

Profile

OpenAIRE Research Graph » History » Version 16

Alessia Bardi, 05/11/2021 03:54 PM

1 1 Alessia Bardi
h1. The OpenAIRE Research Graph
2
3
The OpenAIRE Research Graph is one of the largest open scholarly record collections worldwide, key in fostering Open Science and establishing its practices in the daily research activities. 
4
Conceived as a public and transparent good, populated out of data sources trusted by scientists, the Graph aims at bringing discovery, monitoring, and assessment of science back in the hands of the scientific community.
5
6
Imagine a vast collection of research products all linked together, contextualised and openly available. For the past ten years OpenAIRE has been working to gather this valuable record. It is a massive collection of metadata and links between scientific products such as articles, datasets, software, and other research products, entities like organisations, funders, funding streams, projects, communities, and data sources.
7
8
As of today, the OpenAIRE Research Graph aggregates around 450Mi metadata records with links collecting from 10K data sources trusted by scientists, including:
9
* Repositories registered in OpenDOAR or re3data.org
10
* Open Access journals registered in DOAJ
11
* Crossref
12
* Unpaywall
13
* ORCID
14
* Microsoft Academic Graph
15
* Datacite
16
17
After cleaning, deduplication, enrichment and full-text mining processes, the graph is analysed to produce statistics for OpenAIRE MONITOR (https://monitor.openaire.eu), the Open Science Observatory (https://osobservatory.openaire.eu), made discoverable via OpenAIRE EXPLORE (https://explore.openaire.eu) and programmatically accessible as described at https://develop.openaire.eu. 
18 3 Alessia Bardi
Json dumps are also published on Zenodo.
19 1 Alessia Bardi
20 3 Alessia Bardi
TODO: image of high-level data model (entities and semantic relationships, we can draw here: https://docs.google.com/drawings/d/1c4s7Pk2r9NgV_KXkmX6mwCKBIQ-yK3_m6xsxB-3Km1s/edit)
21
22 1 Alessia Bardi
h2. Graph Data Dumps
23
24 2 Alessia Bardi
In order to facilitate users, different dumps are available. All are available under the "Zenodo community called OpenAIRE Research Graph":https://zenodo.org/communities/openaire-research-graph.
25
Here we provide detailed documentation about the full dump:
26
27
* Json dump: https://doi.org/10.5281/zenodo.3516917
28
* Json schema: https://doi.org/10.5281/zenodo.4238938 
29
30
[[Json schema]]
31
[[FAQ]]
32 1 Alessia Bardi
33
h2. Graph provision processes
34
35 16 Alessia Bardi
[[OpenAIRE entity identifier and PID mapping policy]]
36 10 Alessia Bardi
37
38 11 Alessia Bardi
h3. Aggregation business logic by major sources
39
40
h4. DOIBoost: Crossref, Unpaywall, Microsoft Academic Graph, ORCID
41
42
The idea behind DOIBoost and its origin can be found in the paper (and related resources) at: 
43
44
* La Bruzzo S., Manghi P., Mannocci A. (2019) OpenAIRE's DOIBoost - Boosting CrossRef for Research. In: Manghi P., Candela L., Silvello G. (eds) Digital Libraries: Supporting Open Science. IRCDL 2019. Communications in Computer and Information Science, vol 988. Springer, doi:10.1007/978-3-030-11226-4_11 . Open Access version available at: https://doi.org/10.5281/zenodo.1441071
45
46
In short, the goal is to enrich the records available on Crossref with what's available on Unpaywall, Microsoft Academic Graph, ORCID intersecting all those datasets by DOI.
47
The generation of DOIBoost consists in the following phases:
48
49 13 Alessia Bardi
1 Filter Crossref records that:
50 11 Alessia Bardi
* have blank title
51
* have one of the following publishers: "Test accounts", "CrossRef Test Account"
52
* have no authors with valid names, where valid means: not blank and different from all strings in this list: @List(",", "none none", "none, none", "none &na;", "(:null)", "test test test", "test test", "test", "&na; &na;")@
53
* have "Addie Jackson" as author and "Elsevier BV" as publisher (empirically we say they are test records)
54 12 Alessia Bardi
55 13 Alessia Bardi
2 Intersect Crossref with Unpaywall by DOI (DOIBoost1). The records are enriched with 
56 11 Alessia Bardi
* TODO: AUTHORS?
57
* one @instance@ with 
58
** the @best_oa_location@ of Unpaywall
59
** @color@ set as follows: @green@ if the host is a repository; @gold@ if the host is publisher and the journal is open access; @hybrid@ if the host is publisher, the journal is not open access but there is a license; @bronze@ if no license is available.
60 12 Alessia Bardi
61 13 Alessia Bardi
3 Intersect DOIBoost1 with ORCID (DOIBoost2). The records are enriched with the ORCID identifiers of their authors
62 12 Alessia Bardi
63 13 Alessia Bardi
4 Intersect DOIBoost2 with Microsoft Academic Graph (DOIBoost3). The records are enriched with:
64 11 Alessia Bardi
* abstracts
65
* MAG identifiers of authors
66
* affiliation relationships
67
* subjects (MAG FieldsOfStudy)
68
* conference or journal information (in the @journal@ field) TODO: or @container@, in case of the dump?
69
* [TO BE REMOVED] instances with URL from MAG
70 12 Alessia Bardi
71 13 Alessia Bardi
5 Enrich DOIBoost3 with hosting data sources (@hostedby@) and access right information. In this phase we intersect DOIBoost3 with a dataset composed of journals from OpenAIRE, Crossref, and the ISSN gold list. Each journal comes with its International Standard Serial Numbers (issn, eissn, lissn) and, when available, a flag that tells if the journal is open access. The intersection is done on the basis of the International Standard Serial Numbers. The records with a @journal.[l|e]issn@ that match are enriched as follows:
72 11 Alessia Bardi
* Each instance gain the `hostedby` information. 
73
* If the journal is open access, the access rights of the instances are also set to "Open Access" with "gold" route.
74
75
The hostedby of records that do not match are set to the "Unknown Repository".
76
77 3 Alessia Bardi
h3. Deduplication business logic
78
79
h4. Deduplication business logic for research results 
80
81
Metadata records about the same scholarly work can be collected from different providers. Each metadata record can possibly carry different information because, for example, some providers are not aware of links to projects, keywords or other details. Another common case is when OpenAIRE collects one metadata record from a repository about a pre-print and another record from a journal about the published article. For the provision of statistics, OpenAIRE must identify those cases and “merge” the two metadata records, so that the scholarly work is counted only once in the statistics OpenAIRE produces. 
82
83
Duplicates among research results are identified among results of the same type (publications, datasets, software, other research products). If two duplicate results are aggregated one as a dataset and one as a software, for example, they will never be compared and they will never be identified as duplicates.
84
OpenAIRE supports different deduplication strategies based on the type of results.
85
86
*Methodology overview*
87
88
The deduplication process can be divided into two different phases: 
89
* Candidate identification (clustering)
90 6 Alessia Bardi
* Decision tree
91 10 Alessia Bardi
* Creation of representative record
92 3 Alessia Bardi
93
The implementation of each phase is different based on the type of results that are being processed.
94
95
96
*Strategy for publications*
97
98 1 Alessia Bardi
_Candidate identification (clustering)_
99 6 Alessia Bardi
100 1 Alessia Bardi
Due to the high number of metadata records collected by OpenAIRE, it would not be feasible to compute all possible comparisons between all metadata records.
101 6 Alessia Bardi
The goal of this phase is to limit the number of comparisons by creating groups (or clusters) of records that are likely “similar”. Every record can be added to more than one group. 
102
The decision of inclusion in a group is performed by 2 clustering functions:
103
* Lowercase: doi (in pid list and alternate identifiers list)
104
* WordsStatsSuffixPrefixChain: suffixprefix with statistics on the full title (number_of_words & number_of_letters%10)
105
Example: 
106
If title is : “Search for the Standard Model Higgs Boson”
107 3 Alessia Bardi
The clustering function produces 2 keys (i.e. adds the publication to two clusters): [5-3-seaardmod, 5-3-rchstadel]
108
109 6 Alessia Bardi
110
_Desicision tree_
111
112
For each pair of publications in a cluster the following strategy (depicted in the figure below) is applied.
113
Cross comparison of the pid lists (in the @pid@ and @alternateid@ elements). If 50% common pids, levenshtein distance on titles with low threshold (0.9).
114
Otherwise, check if the number of authors and the title version is equal. If so, levenshtein distance on titles with higher threshold (0.99). 
115 3 Alessia Bardi
The publications are matched as duplicate if the distance is higher than the threshold, in every other case they are considered as distinct publications.
116
117
!dedup-results.png!
118
119 10 Alessia Bardi
_Creation of representative record_
120
121
TODO
122
123
124 3 Alessia Bardi
*Strategy for datasets*
125
126
*Strategy for software*
127
128
*Strategy for other types of research products*
129
130 8 Alessia Bardi
*Clustering functions*
131 3 Alessia Bardi
132
_NgramPairs_
133
It produces a list of concatenations of a pair of ngrams generated from different words.
134
Example:
135
Input string: “Search for the Standard Model Higgs Boson”
136 1 Alessia Bardi
Parameters: ngram length = 3
137 8 Alessia Bardi
List of ngrams: “sea”, “sta”, “mod”, “hig”
138
Ngram pairs: “seasta”, “stamod”, “modhig”
139 3 Alessia Bardi
140
_SuffixPrefix_
141
It produces ngrams pairs in a particular way: it concatenates the suffix of a string with the prefix of the next in the input string.
142
Example:
143
Input string: “Search for the Standard Model Higgs Boson”
144 1 Alessia Bardi
Parameters: suffix and prefix length = 3
145 3 Alessia Bardi
Output list: “ardmod” (suffix of the word “Standard” + prefix of the word “Model”), “rchsta” (suffix of the word “Search” + prefix of the word “Standard”)
146 1 Alessia Bardi
147
h3. TODOs
148
149 10 Alessia Bardi
* OpenAIRE entity identifier & PID mapping policy (started, to be completed by Claudio and/or Michele DB)
150 1 Alessia Bardi
* Aggregation business logic by major sources:
151 15 Alessia Bardi
** -Unpaywall integration-
152
** -Crossref integration- 
153
** -ORCID integration-
154
** -Cross cleaning actions: hostedBy patch-
155 1 Alessia Bardi
** Scholexplorer business logic (relationship resolution)
156 3 Alessia Bardi
** DataCite
157 1 Alessia Bardi
** EuropePMC
158
** more….
159 10 Alessia Bardi
* Deduplication business logic (started, to be completed by Michele DB)
160 9 Alessia Bardi
** For research outputs ( -publications- , datasets, software, orp)
161 1 Alessia Bardi
** For research organizations 
162
* Enrichment
163
** Mining business logic
164
** Deduction-based inference 
165
** Propagation business logic
166
* Post-cleaning business logic
167
* FAQ