Project

General

Profile

OpenAIRE Research Graph » History » Version 31

Claudio Atzori, 10/11/2021 02:21 PM

1 1 Alessia Bardi
h1. The OpenAIRE Research Graph
2
3 25 Claudio Atzori
{{>toc}}
4
5 1 Alessia Bardi
The OpenAIRE Research Graph is one of the largest open scholarly record collections worldwide, key in fostering Open Science and establishing its practices in the daily research activities. 
6
Conceived as a public and transparent good, populated out of data sources trusted by scientists, the Graph aims at bringing discovery, monitoring, and assessment of science back in the hands of the scientific community.
7
8
Imagine a vast collection of research products all linked together, contextualised and openly available. For the past ten years OpenAIRE has been working to gather this valuable record. It is a massive collection of metadata and links between scientific products such as articles, datasets, software, and other research products, entities like organisations, funders, funding streams, projects, communities, and data sources.
9
10
As of today, the OpenAIRE Research Graph aggregates around 450Mi metadata records with links collecting from 10K data sources trusted by scientists, including:
11
* Repositories registered in OpenDOAR or re3data.org
12
* Open Access journals registered in DOAJ
13 30 Claudio Atzori
* [[DOIBoost#Inputs|Crossref]]
14
* [[DOIBoost#Inputs|Unpaywall]]
15
* [[DOIBoost#Inputs|ORCID]]
16
* [[DOIBoost#Inputs|Microsoft Academic Graph]]
17 1 Alessia Bardi
* Datacite
18
19
After cleaning, deduplication, enrichment and full-text mining processes, the graph is analysed to produce statistics for OpenAIRE MONITOR (https://monitor.openaire.eu), the Open Science Observatory (https://osobservatory.openaire.eu), made discoverable via OpenAIRE EXPLORE (https://explore.openaire.eu) and programmatically accessible as described at https://develop.openaire.eu. 
20 3 Alessia Bardi
Json dumps are also published on Zenodo.
21 1 Alessia Bardi
22 3 Alessia Bardi
TODO: image of high-level data model (entities and semantic relationships, we can draw here: https://docs.google.com/drawings/d/1c4s7Pk2r9NgV_KXkmX6mwCKBIQ-yK3_m6xsxB-3Km1s/edit)
23
24 1 Alessia Bardi
h2. Graph Data Dumps
25
26 2 Alessia Bardi
In order to facilitate users, different dumps are available. All are available under the "Zenodo community called OpenAIRE Research Graph":https://zenodo.org/communities/openaire-research-graph.
27
Here we provide detailed documentation about the full dump:
28
29
* Json dump: https://doi.org/10.5281/zenodo.3516917
30
* Json schema: https://doi.org/10.5281/zenodo.4238938 
31
32 26 Claudio Atzori
h3. [[Json schema]]
33
34
h3. [[FAQ]]
35 1 Alessia Bardi
36
h2. Graph provision processes
37
38 29 Alessia Bardi
TODO:
39
3. Processes
40
* harvesting NOT SURE WHAT IOANNA WANTS: is what we have on graph.openaire.eu OK?
41
* transformation NOT SURE WHAT IOANNA WANTS: is what we have on graph.openaire.eu OK?
42
* -doiboost- (not in processing, in [[#Aggregation business logic by major sources]])
43
* direct zenodo updates
44
* deduplication STARTED
45
* inference
46
* Funder ingestion ("Harry is speaking with funders, gets the list of projects, inference rules, ...")
47
For the processes we need a description of what it does, what's the input, which part of the graph it affects, and anything of importance ("if there is no input value for X, then we assume Y and assign the value Z to A")
48
49 16 Alessia Bardi
[[OpenAIRE entity identifier and PID mapping policy]]
50 10 Alessia Bardi
51 11 Alessia Bardi
h3. Aggregation business logic by major sources
52
53 28 Alessia Bardi
TODO:
54
2. input sources
55
* repositories
56
* journals
57
* -DOIBoost-
58
* * -MAG-
59
* * -Crossref-
60
* * -Unpaywall-
61
* projects
62
* organizations
63
* openorgs
64
* open citations
65
* openAPC
66
* etc etc
67
68
For each input source class, we need to know what's in there, the format, approximate numbers, peculiarities, important aspects of the data (e.g. "Crossref provides us with the authoritative list of publishers!"). Anything else of importance?
69
70
71 20 Alessia Bardi
DOIBoost is the intersection among Crossref, Unpaywall, Microsoft Academic Graph and ORCID
72 1 Alessia Bardi
73 27 Claudio Atzori
h4. [[DOIBoost]]
74 1 Alessia Bardi
75 27 Claudio Atzori
h4. [[Datacite]]
76 11 Alessia Bardi
77 27 Claudio Atzori
h4. [[EuropePMC]]
78
79 21 Alessia Bardi
The strategy for the resolution of links between publications and datasets is defined by Scholexplorer
80 27 Claudio Atzori
81 21 Alessia Bardi
[[Scholexplorer]]
82
83 3 Alessia Bardi
h3. Deduplication business logic
84
85
h4. Deduplication business logic for research results 
86
87
Metadata records about the same scholarly work can be collected from different providers. Each metadata record can possibly carry different information because, for example, some providers are not aware of links to projects, keywords or other details. Another common case is when OpenAIRE collects one metadata record from a repository about a pre-print and another record from a journal about the published article. For the provision of statistics, OpenAIRE must identify those cases and “merge” the two metadata records, so that the scholarly work is counted only once in the statistics OpenAIRE produces. 
88
89
Duplicates among research results are identified among results of the same type (publications, datasets, software, other research products). If two duplicate results are aggregated one as a dataset and one as a software, for example, they will never be compared and they will never be identified as duplicates.
90
OpenAIRE supports different deduplication strategies based on the type of results.
91
92
*Methodology overview*
93
94
The deduplication process can be divided into two different phases: 
95
* Candidate identification (clustering)
96 6 Alessia Bardi
* Decision tree
97 10 Alessia Bardi
* Creation of representative record
98 3 Alessia Bardi
99
The implementation of each phase is different based on the type of results that are being processed.
100
101
102
*Strategy for publications*
103
104 1 Alessia Bardi
_Candidate identification (clustering)_
105 6 Alessia Bardi
106
107 22 Paolo Manghi
Clustering is a common heuristics used to overcome the N x N complexity required to match all pairs of objects to identify the equivalent ones. The challenge is to identify a clustering function that maximizes the chance of comparing only records that may lead to a match, while minimizing the number of records that will not be matched while being equivalent. Since the equivalence function is to some level tolerant to minimal errors (e.g. switching of characters in the title, or minimal difference in letters), we need this function to be not too precise (e.g. a hash of the title), but also not too flexible (e.g. random ngrams of the title). On the other hand, reality tells us that in some cases equality of two records can only be determined by their PIDs (e.g. DOI) as the metadata properties are very different across different versions and no clustering function will ever bring them into the same cluster. To match these requirements OpenAIRE clustering for products works with two functions:
108
* DOI: the function generates the DOI when this is provided as part of the record properties;
109
* Title-based function: the function generates a key that depends on (i) number of significant words in the title (normalized, stemming, etc.), (ii) module 10 of the number of characters of such words, and (iii) a string obtained as an alternation of the function prefix(3) and suffix(3) (and vice versa) o the first 3 words (2 words if the title only has 2). For example, the title “Entity deduplication in big data graphs for scholarly communication” becomes “entity deduplication big data graphs scholarly communication” with two keys key “7.1entionbig” and “7.1itydedbig” (where 1 is module 10 of 54 characters of the normalized title.
110 6 Alessia Bardi
111 22 Paolo Manghi
_Decision tree_
112 6 Alessia Bardi
113
For each pair of publications in a cluster the following strategy (depicted in the figure below) is applied.
114
Cross comparison of the pid lists (in the @pid@ and @alternateid@ elements). If 50% common pids, levenshtein distance on titles with low threshold (0.9).
115
Otherwise, check if the number of authors and the title version is equal. If so, levenshtein distance on titles with higher threshold (0.99). 
116 3 Alessia Bardi
The publications are matched as duplicate if the distance is higher than the threshold, in every other case they are considered as distinct publications.
117
118 24 Claudio Atzori
!{width: 50%}dedup-results.png!
119 3 Alessia Bardi
120 10 Alessia Bardi
_Creation of representative record_
121
122
TODO
123
124
125 3 Alessia Bardi
*Strategy for datasets*
126
127
*Strategy for software*
128
129
*Strategy for other types of research products*
130
131 8 Alessia Bardi
*Clustering functions*
132 3 Alessia Bardi
133
_NgramPairs_
134
It produces a list of concatenations of a pair of ngrams generated from different words.
135
Example:
136 31 Claudio Atzori
Input string: @“Search for the Standard Model Higgs Boson”@
137 1 Alessia Bardi
Parameters: ngram length = 3
138 31 Claudio Atzori
List of ngrams: @“sea”, “sta”, “mod”, “hig”@
139
Ngram pairs: @“seasta”, “stamod”, “modhig”@
140 3 Alessia Bardi
141
_SuffixPrefix_
142
It produces ngrams pairs in a particular way: it concatenates the suffix of a string with the prefix of the next in the input string.
143
Example:
144 31 Claudio Atzori
Input string: @“Search for the Standard Model Higgs Boson”@
145 1 Alessia Bardi
Parameters: suffix and prefix length = 3
146 31 Claudio Atzori
Output list: @“ardmod”@ (suffix of the word @“Standard”@ + prefix of the word @“Model”@), @“rchsta”@ (suffix of the word @“Search”@ + prefix of the word @“Standard”@)
147 1 Alessia Bardi
148 25 Claudio Atzori
h2. TODOs
149 1 Alessia Bardi
150 10 Alessia Bardi
* OpenAIRE entity identifier & PID mapping policy (started, to be completed by Claudio and/or Michele DB)
151 1 Alessia Bardi
* Aggregation business logic by major sources:
152 15 Alessia Bardi
** -Unpaywall integration-
153
** -Crossref integration- 
154
** -ORCID integration-
155
** -Cross cleaning actions: hostedBy patch-
156 1 Alessia Bardi
** Scholexplorer business logic (relationship resolution)
157 3 Alessia Bardi
** DataCite
158 1 Alessia Bardi
** EuropePMC
159
** more….
160 10 Alessia Bardi
* Deduplication business logic (started, to be completed by Michele DB)
161 9 Alessia Bardi
** For research outputs ( -publications- , datasets, software, orp)
162 1 Alessia Bardi
** For research organizations 
163
* Enrichment
164
** Mining business logic
165
** Deduction-based inference 
166
** Propagation business logic
167
* Post-cleaning business logic
168
* FAQ