Project

General

Profile

DOIBoost » History » Version 49

Claudio Atzori, 11/03/2022 04:21 PM

1 1 Alessia Bardi
h1. DOIBoost
2
3 31 Claudio Atzori
{{>toc}}
4
5 4 Alessia Bardi
h2. DOIBoost: Crossref, Unpaywall, Microsoft Academic Graph, ORCID
6 1 Alessia Bardi
7
The idea behind DOIBoost and its origin can be found in the paper (and related resources) at: 
8
9
* La Bruzzo S., Manghi P., Mannocci A. (2019) OpenAIRE's DOIBoost - Boosting CrossRef for Research. In: Manghi P., Candela L., Silvello G. (eds) Digital Libraries: Supporting Open Science. IRCDL 2019. Communications in Computer and Information Science, vol 988. Springer, doi:10.1007/978-3-030-11226-4_11 . Open Access version available at: https://doi.org/10.5281/zenodo.1441071
10
11 42 Claudio Atzori
In short, the goal is to enrich the records available on Crossref with what's available on Unpaywall, Microsoft Academic Graph, ORCID intersecting all those datasets by DOI. As consequence, DOIBoost does not contain any record from MAG, Unpaywall, or ORCID that doesn't provide a DOI available in Crossref.
12 5 Alessia Bardi
13 43 Alessia Bardi
Each Crossref record is enriched with:
14
* ORCID identifiers of authors from ORCID
15
* Open Access instance (with OA color/route and license) from Unpaywall
16
* the following information from MAG:
17
** abstracts
18
** MAG identifiers of authors
19 48 Claudio Atzori
** affiliation (result - organization) relationships
20 43 Alessia Bardi
** subjects (MAG FieldsOfStudy)
21
** conference or journal information 
22
23 44 Alessia Bardi
The Open Access status is also set by intersecting the journal information of a record with the journal lists available from DOAJ and the Gold ISSN list.
24
25 43 Alessia Bardi
26 5 Alessia Bardi
h3. Inputs
27
28 6 Alessia Bardi
* *Crossref*: dump available to Crossref subscribers via MetadataPlus service, updated once a month.
29 27 Claudio Atzori
* *Microsoft Academic Graph*: downloaded version on 2021-02-15. We plan to take the latest version in Dec 2021 before MAG will be retired.
30 18 Alessia Bardi
* *ORCID*: baseline dump obtained in 2020-10-13, regularly updated every week from the "ORCID public API":https://info.orcid.org/documentation/features/public-api/
31 7 Alessia Bardi
* *Unpaywall*: public database snapshot downloaded in March 2021. Unpaywall updates it twice a year (https://unpaywall.org/products/snapshot) 
32 5 Alessia Bardi
33 27 Claudio Atzori
The construction of the DOIBoost dataset consists of the following phases:
34 1 Alessia Bardi
35 26 Claudio Atzori
h3. 1 Filtering
36 1 Alessia Bardi
37 26 Claudio Atzori
Records in Crossref are ruled out according to the following criteria
38
39 1 Alessia Bardi
* have blank title
40 26 Claudio Atzori
* have one of the following publishers: @"Test accounts"@, @"CrossRef Test Account"@
41 49 Claudio Atzori
** Examples from https://api.crossref.org/works?query.publisher-name=%22Test%20accounts%22
42
*** 10.1007/bf00344543
43
*** 10.1007/bf00186154
44
*** 10.1306/64ed947a-1724-11d7-8645000102c1865d
45 1 Alessia Bardi
* have no authors with valid names, where valid means: not blank and different from all strings in this list: @List(",", "none none", "none, none", "none &na;", "(:null)", "test test test", "test test", "test", "&na; &na;")@
46 26 Claudio Atzori
* have @"Addie Jackson"@ as author and @"Elsevier BV"@ as publisher (empirically we say they are test records)
47 49 Claudio Atzori
** Examples from https://api.crossref.org/works?query.author=Addie+Jackson&query.publisher-name=%22Elsevier%20BV%22
48
*** 10.2139/ssrn.2082156
49
*** 10.2139/ssrn.2202300
50
*** 10.2139/ssrn.2255657
51 15 Alessia Bardi
* have not one of the following values in the field @type@:
52 26 Claudio Atzori
**   @"book-section"@
53
**   @"book"@
54
**   @"book-chapter"@
55
**   @"book-part"@
56
**   @"book-series"@
57
**   @"book-set"@
58
**   @"book-track"@
59
**   @"edited-book"@
60
**   @"reference-book"@
61
**   @"monograph"@
62
**   @"journal-article"@
63
**   @"dissertation"@
64
**   @"other"@
65
**   @"peer-review"@
66
**   @"proceedings"@
67
**   @"proceedings-article"@
68
**   @"reference-entry"@
69
**   @"report"@
70
**   @"report-series"@
71
**   @"standard"@
72
**   @"standard-series"@
73
**   @"posted-content"@
74
**   @"dataset"@
75 15 Alessia Bardi
76 1 Alessia Bardi
Records with @type=dataset@ are mapped into OpenAIRE results of type dataset. All others are mapped as OpenAIRE results of type publication.
77 19 Alessia Bardi
78 30 Claudio Atzori
h3. 2 Mapping Crossref properties into the OpenAIRE Research Graph
79 16 Alessia Bardi
80 27 Claudio Atzori
Properties in OpenAIRE results are set based on the logic described in the following table:
81 16 Alessia Bardi
82 1 Alessia Bardi
|_.OpenAIRE Result field path|_.Crossref path(s)|_.Notes|
83 33 Alessia Bardi
| id | doi | id in the form @doi_________::md5(doi) |
84 34 Alessia Bardi
| dateofcollection|  indexed.datetime|  |
85
| lastupdatetimestamp | indexed.timestamp | |
86 1 Alessia Bardi
| type | type | @dataset@ if the Crossref type is dataset, @publication@ otherwise (based on the logics [[DOIBoost#1-Filtering|described above]]) |
87 34 Alessia Bardi
| originalId | doi, clinical-trial-number, alternative-id | |
88
| pid |  | The scheme tells the type of PID, the value contains the actual value |
89
| pid.scheme |  | Default value: doi |
90
| pid.value | doi | The doi is normalised and lower-cased|
91 1 Alessia Bardi
| maintitle | title |  |
92
| subtitle | subtitle |  |
93 37 Alessia Bardi
| author | author | if available the sequence is mapped to rank and the ORCID is also mapped |
94
| author.name | author.given |  |
95
| author.surname | author.family |  |
96
| author.fullname | author.given author.family|  |
97
| author.rank | | based on the order, starts from 1 |
98
| author.pid | | only if the ORCID is available |
99
| author.pid.id.scheme | | Default 'pending_orcid' (meaning that it is not an id confirmed by ORCID |
100
| author.pid.id.value | author.ORCID |  |
101
| author.pid.provenance.provenance| | Default 'Harvested' |
102
| author.pid.provenance.trust| | Default '0.9' |
103 1 Alessia Bardi
| description | abstract |  |
104 34 Alessia Bardi
| subject | subject | with classid='keywords', i.e. no controlled vocabularies for Crossref subjects |
105
| publicationdate | issued.datetime or, if not available, created.datetime |  |
106
| publisher | publisher |  |
107
| source | source |  only if the record is not of type @book@ |
108
| source | concatenation of @container-title.head@ "ISBN: " @ISBN.head@ | only if the record is of type @book@ |
109
| container |  | It is set only for publications with information about the journal it was published in. |
110
| container.name | container-title.head| |
111
| container.issnOnline | issn-type.value| if issn-type.type='electronic'|
112
| container.issnPrinted | issn-type.value| if issn-type.type='print'|
113
| container.vol | volume| |
114
| container.sp | page | before '-'| 
115
| container.ep | page | after '-'| 
116
| instance | | One instance is created with the DOI URL|
117 1 Alessia Bardi
| instance.accessright |  | Values in @instance.accessright.code@ and @instance.accessright.label@ are set based on license and dateofacceptance: 
118
- UNKNOWN: if the license is blank
119
- OPEN ACCESS: if the license is a CC license or an ACS license or an APA license (considered OPEN also by Unpaywall, see "Unpaywall FAQ":https://support.unpaywall.org/support/solutions/articles/44002063718-what-is-an-oa-license- for details) or if OUP license, but only after 12 months from the publication date
120
- EMBARGO: OUP license, before 12 months from the publication date
121
- CLOSED: if there is a license not covered by the previous cases |
122 33 Alessia Bardi
| instance.accessright.code |  | Code from the "COAR vocabulary for access right":http://vocabularies.coar-repositories.org/documentation/access_rights/|
123
| instance.accessright.label |  | One of: OPEN, RESTRICTED, CLOSED, EMBARGO |
124
| instance.accessright.scheme |  | Scheme that defines the code and label, i.e. the URL to the "COAR vocabulary for access right":http://vocabularies.coar-repositories.org/documentation/access_rights/|
125
| instance.accessright.openAccessRoute |  | only if instance.accessright.value = 'OPEN ACCESS'. Default is 'hybrid'. The route is fixed in subsequent phases of DOIBoost, namely when intersecting with Unpaywall and patching the hostedby via DOAJ and the Gold-ISSN list.|
126
| instance.license | license.URL| If there is a @license.content-version='vor'@, then this is used. Otherwise the first license entry is used. |
127 34 Alessia Bardi
| instance.pid |  | The scheme tells the type of PID, the value contains the actual value |
128
| instance.pid.scheme |  | Default value: doi  |
129
| instance.pid.value | doi | The doi is normalised and lower-cased  |
130
| instance.publicationdate | issued.datetime or, if not available, created.datetime|  |
131 40 Alessia Bardi
| instance.refereed |  | set to 'peerReviewed' only if @relation.has-review.id@ is not empty. UNKNOWN otherwise.|
132 33 Alessia Bardi
| instance.type | subtype | mapped using the "OpenAIRE vocabulary for result typologies":https://api.openaire.eu/vocabularies/dnet:result_typologies |
133 34 Alessia Bardi
| instance.url | doi| Full URL of the DOI | 
134 33 Alessia Bardi
135
All other fields of the [[Json_schema]] not mentioned in the table contain empty values.
136
137
All the records from Crossref are related to the datasource with @name=Crossref@ and @id=openaire____::081b82f96300b6a6e3d282bad31cb6e2@
138
139 34 Alessia Bardi
Possible improvements:
140
* map @clinical-trial-number@ and @alternative-id@ in alternateIdentifiers?
141
* Verify if Crossref has a property for @language@, @country@, @container.issnLinking@, @container.iss@, @container.edition@, @container.conferenceplace@ and @container.conferencedate@ 
142
* Different approach to set the @refereed@ field and improve its coverage?
143 2 Alessia Bardi
144 4 Alessia Bardi
h3. 2 Map Crossref links to projects/funders
145
146 3 Alessia Bardi
Links to funding available in Crossref are mapped as funding relationships (@result@ -- @isProducedBy@ --> @project@) applying the following mapping:
147
148
| *funder* | *grant code* | *Link to* |
149
| DOI: {10.13039/100010663, 10.13039/100010661, 10.13039/501100007601, 10.13039/501100000780, 10.13039/100010665}
150
or name: 'European Union’s Horizon 2020 research and innovation program' | series of 4-9 digits in @award@ | Link to H2020 project |
151
| DOI: {10.13039/100011199, 10.13039/100004431, 10.13039/501100004963, 10.13039/501100000780} | series of 4-9 digits in @award@  | Link to FP7 project |
152
| DOI: 10.13039/501100000781 OR name: 'European Union's'| series of 4-9 digits in @award@ | Link to FP7 or H2020 project |
153
| DOI: 10.13039/100000001 | @award@ | Link to NSF project |
154
| DOI: 10.13039/501100001665 OR name: {'The French National Research Agency (ANR)', 'The French National Research Agency'} | @award@ | Link to ANR project |
155
| DOI: 10.13039/501100002341 | @award@ | Link to Academy of Finland project |
156
| DOI: 10.13039/501100001602 | @award@, removing the initial 'SFI' if present | Link to SFI project |
157
| DOI: 10.13039/501100000923 | @award@ | Link to ARC project |
158
| DOI: 10.13039/501100000038 | @award@ ignore: we cannot map the project codes in Crossref to project codes in OpenAIRE | Link to NSERC (@unidentified@ project) |
159
| DOI: 10.13039/501100000155 | @award@ ignore: we cannot map the project codes in Crossref to project codes in OpenAIRE | Link to SSHRC (@unidentified@ project) |
160
| DOI: 10.13039/501100000024 | @award@ ignore: we cannot map the project codes in Crossref to project codes in OpenAIRE | Link to CIHR (@unidentified@ project) |
161
| DOI: 10.13039/501100002848 OR name :'CONICYT, Programa de Formación de Capital Humano Avanzado' | @award@ | Link to CONICYT project |
162
| DOI: 10.13039/501100003448 | series of 4-9 digits in @award@ | Link to GSRT project |
163
| DOI: 10.13039/501100010198 | @award@ | Link to SGOV project |
164
| DOI: 10.13039/501100004564 | series of 4-9 digits in @award@ | Link to MESTD project |
165
| DOI: 10.13039/501100003407 | @award@ | Link to MIUR project. Since OpenAIRE has a small subset of MIUR projects, a link to the MIUR funder (@unidentified@ project) is also generated |
166
| DOI: {10.13039/501100006588, 10.13039/501100004488} | @award@, removing 'Project No' and 'HRZZ' prefix, if present | Link to HRZZ or MZOS project |
167
| DOI: 10.13039/501100006769 | @award@ | Link to Russian Science Foundation project |
168 1 Alessia Bardi
| DOI: 10.13039/501100001711 | @award@ after '_' and before '/' | Link to SNSF project |
169
| DOI: 10.13039/501100004410 | @award@ | Link to TUBITAK project |
170 4 Alessia Bardi
| DOI: 10.10.13039/100004440  or name: 'Wellcome Trust Masters Fellowship'| @award@ | Link to Wellcome Trust specific project and to the @unidentified@ project.|
171 1 Alessia Bardi
172 36 Alessia Bardi
h3. 3 Intersect Crossref with UnpayWall by DOI (DOIBoost1)
173 10 Alessia Bardi
174 36 Alessia Bardi
The fields we consider from UnpayWall are:
175 10 Alessia Bardi
* @is_oa@
176
* @best_oa_location@
177
* @oa_status@
178 1 Alessia Bardi
179 36 Alessia Bardi
The results of Crossref that intersect by DOI with UnpayWall records are enriched with one additional @instance@ with the following properties:
180 1 Alessia Bardi
181
|_.OpenAIRE Result field path |_.Unpaywall field path |_.Notes |
182 36 Alessia Bardi
| instance | | created only if @is_oa@ and a @best_oa_location@ is available |
183
| instance.accessright | | default value  Open Access: we do not add instances if UnpayWall says there is no open version|
184
| instance.accessright.code |  | Open Access code from the "COAR vocabulary for access right":http://vocabularies.coar-repositories.org/documentation/access_rights/|
185
| instance.accessright.label |  | Always OPEN |
186
| instance.accessright.scheme |  | Scheme that defines the code and label, i.e. the URL to the "COAR vocabulary for access right":http://vocabularies.coar-repositories.org/documentation/access_rights/|
187
| instance.accessright.openAccessRoute | @oa_status@ | |
188
| instance.url | @best_oa_location@ | |
189
| instance.license | @best_oa_location.license@ | |
190
| instance.pid |  | The scheme tells the type of PID, the value contains the actual value |
191
| instance.pid.scheme |  | Default value: doi  |
192
| instance.pid.value | doi | The doi is normalised and lower-cased  |
193 10 Alessia Bardi
194 36 Alessia Bardi
For the definition of UnpayWall's @oa_status@ refer to the "Unpaywall FAQ":https://support.unpaywall.org/support/solutions/articles/44001777288-what-do-the-types-of-oa-status-green-gold-hybrid-and-bronze-mean-
195
196
The record will also feature a relation to the UnpayWall data source: @name="UnpayWall"@, @id=openaire____::8ac8380272269217cb09a928c8caa993@.
197 9 Alessia Bardi
198 21 Alessia Bardi
h3. 4 Intersect DOIBoost1 with ORCID (DOIBoost2)
199
200 23 Alessia Bardi
The fields we consider from ORCID are:
201 21 Alessia Bardi
* @doi@
202
* @authors@, a list of authors, each with optional @name@, @surname@, @creditName@, @oid@
203
204
|_.OpenAIRE field path|_.ORCID path|_.Notes|
205
| pid | doi  |  |
206
| author.name | capitalize(name) |  only mapped if not blank|
207
| author.surname | capitalize(surname) | only mapped if not blank |
208
| author.fullname |  | if name and surname are not blank, they are concatenated (capitalize(name) capitalize(surname)), otherwise we use the creditName  |
209 38 Alessia Bardi
| author.pid | | only if the ORCID is available |
210
| author.pid.id.scheme | | Default 'orcid' (meaning that it is confirmed by ORCID, (in contrast to the 'orcid_pending' set from Crossref and Unpaywall) |
211
| author.pid.id.value | oid |  |
212
| author.pid.provenance.provenance| | Default 'Harvested' |
213
| author.pid.provenance.trust| | Default '0.9' |
214 1 Alessia Bardi
215 4 Alessia Bardi
The records are enriched with the ORCID identifiers of their authors.
216 46 Alessia Bardi
217
TODO: Update with the new approach implemented by Miriam. 
218
219 38 Alessia Bardi
The current approach is: 
220 1 Alessia Bardi
* if the number of authors from Crossref equals the size of authors from ORCID, then we pick the list of authors with more PIDs and try to enrich it with the PIDs from the other list, based on JaroWrinkler distance on on authors' names, surnames, or fullnames, depending on which properties are available;
221
* if the number of authors are different, then we take the longest and try to enrich it with the PIDs from the other author list, based on JaroWrinkler distance on on authors' names, surnames, or fullnames, depending on which properties are available
222
223 38 Alessia Bardi
Miriam will modify the process to ensure that:
224
* the list of authors from Crossred always "win"
225
* the identifiers from ORCID "win"
226 46 Alessia Bardi
227 24 Alessia Bardi
h3. 5 Intersect DOIBoost2 with Microsoft Academic Graph (DOIBoost3)
228
229
*Important Notes*
230
* Only papers with DOI are considered
231
* Since for the same DOI we have multiple version of item with different MAG PaperId, we only take one per DOI (the last one we process). We call this dataset @Papers_distinct@
232
233
When mapping MAG records to the OpenAIRE Research Graph, we consider the the following MAG tables:
234
* @PaperAbstractsInvertedIndex@: for the paper abstracts
235
* @Authors@: for the authors. The MAG data is pre-processed by grouping authors by PaperId 
236
* @Affiliations@ and @PaperAuthorAffiliations@: to generate links between publications and organisations
237
* @Journals@ and @ConferenceInstances@: joined with @Papers_distinct@ to have the information about the venues where the paper was published
238
* TO BE REMOVED @PaperUrls@: to create one instance for the OpenAIRE publication
239 1 Alessia Bardi
* @FieldsOfStudy@: to add subjects
240
241
The records are enriched with:
242
* abstracts
243
* MAG identifiers of authors
244 4 Alessia Bardi
* affiliation relationships
245
* subjects (MAG FieldsOfStudy)
246 24 Alessia Bardi
* conference or journal information (in the @journal@ field) TODO: or @container@, in case of the dump?
247
* [TO BE REMOVED] instances with URL from MAG
248
249
250
TODO: ensure we use the field names of the public dump
251
252
|_.OpenAIRE path| ._MAG table |_.MAG path(s)|_.Notes|
253
| pid | Papers_distinct | Doi | |
254 39 Alessia Bardi
| pid.scheme |  | | Default value: doi |
255
| pid.value | | doi | The doi is normalised and lower-cased|
256 24 Alessia Bardi
| originalId | Papers_distinct | PaperId | |
257
| title | Papers_distinct | PaperTitle | as main title |
258
| title | Papers_distinct | OriginalTitle | as alternative title |
259
| source | Papers_distinct | BookTitle |  |
260
| dateofacceptance | Papers_distinct  | Date |  first 10 chars, if not blank |
261 1 Alessia Bardi
| publisher | Papers_distinct, Journal  | Publisher |   |
262
| description | PaperAbstractsInvertedIndex | IndexedAbstract | |
263 47 Alessia Bardi
| container | ConferenceInstances, Journals |  ||
264
| container.name | | DisplayName ||
265
| container.conferencePlace | | Location ||
266
| container.conferenceDate | | StartDate and EndDate | Values created as concatanation of the first 10 chars of each (separated by '-'), if both are not blank|
267
| container.sp | | FirstPage ||
268
| container.ep | | EndPage ||
269
| container.issnPrinted | | Issn ||
270
| container.vol | Papers_distinct | Volume ||
271
| container.iss | Papers_distinct | Issue ||
272 24 Alessia Bardi
| subject |FieldsOfStudy  | subjects | All subjects from MAG are set with a dedicated marker in the classname/classid 'Microsoft Academic Graph Classification'/MAG. 
273
We create one subject per DisplayName, per MainType and, if the MainType is in the format @x.y@, one subject also for the first token (i.e. @x@) |
274
| subject.value | | subjects.DisplayName, subjects.MainType, split(subjects.MainType, '.'0.head | All subject from MAG are set with a dedicated marker in the classname/classid 'Microsoft Academic Graph Classification'/MAG|
275
| author | Authors, PaperAuthorAffiliations | |  |
276
| author.rank |  | sequenceNumber |  |
277
| author.fullName |  | DisplayName | if not blank |
278
| author.affiliation |  | affiliation | if not null |
279
| author.pid |  | AuthorId |  MAG id of the author as URL. TO BE REMOVED? |
280
| instance | PaperUrls | | TO BE REMOVED. Currently maps to the MAG URl and to any URLs in SourceUrl |
281 4 Alessia Bardi
282
h3. 6 Enrich DOIBoost3 with hosting data sources (@hostedby@) and access right information
283 27 Claudio Atzori
284 25 Alessia Bardi
In this phase, we intersect DOIBoost3 with a dataset composed of journals from OpenAIRE, Crossref, and the ISSN gold list. Each journal comes with its International Standard Serial Numbers (issn, eissn, lissn) and, when available, a flag that tells if the journal is open access. The intersection is done on the basis of the International Standard Serial Numbers. The records with a @journal.[l|e]issn@ that match are enriched as follows:
285 1 Alessia Bardi
* Each instance gain the @hostedby@ information corresponding to the journal
286
* If the journal is open access, the access rights of the instances are also set to "Open Access" with "gold" route (because by construction, the journals we know are open are from DOAJ or Gold ISSN list)
287
288
The hostedby of records that do not match are set to the "Unknown Repository".