Project

General

Profile

D64 OpenAIRE Maintenance Report » History » Version 80

Marek Horst, 28/06/2018 12:33 PM
Updating Information Space content status as of May 2018

1 78 Marek Horst
h1. D6.4 OpenAIRE Maintenance Report (v2, 28th of June 2018)
2 1 Marek Horst
3
{{toc}}
4
5
h2. Overview
6
7 65 Marek Horst
This document contains information about the status of OpenAIRE2020 workflows, services and content. It provides details about deployment and history of major modifications of the system hosted at ICM, Poland and Zenodo repository hosted at CERN, Switzerland. The official maintenance of the OpenAIRE2020 services began on January 1st, 2015 when the project started.
8 1 Marek Horst
9 70 Marek Horst
_Information Space Data_ section focuses on OpenAIRE data model including stats for main entities, inferred relations, funders, available data sources and records stored in Zenodo repository. 
10 1 Marek Horst
11 70 Marek Horst
_OpenAIRE workflows_ describes in depth all subsequent phases of data processing:
12 65 Marek Horst
* aggregation
13
* Information Space population
14
* deduplication
15
* inference generation
16 67 Alessia Bardi
* Information Space monitoring and publishing
17 65 Marek Horst
18 67 Alessia Bardi
The last section gives insight into software life-cycle of all major services involved in the project, whole infrastructure and most important architectural changes.
19 1 Marek Horst
20 69 Marek Horst
h2. Information Space Data
21 1 Marek Horst
22 53 Marek Horst
The OpenAIRE [[Core Data Model]] comprises the following interlinked entities: 
23
* results ( in form of publications, datasets and patents)
24 79 Marek Horst
* -persons- (removed since 2018.01)
25 53 Marek Horst
* organisations
26
* funders
27
* funding streams
28
* projects
29
* data sources (in form of institutional, thematic, and data repositories, Current Research Information Systems (CRIS), thematic and national aggregators, publication catalogues and entity registries)
30 43 Jochen Schirrwagen
31 56 Marek Horst
h3. Content Status
32
33
First table presents numbers for different InformationSpace main entities and fulltexts:
34
35
|*data type*|*count*|
36 80 Marek Horst
|publication metadata|24629933|
37
|dataset metadata|7389872|
38
|projects|2539451|
39
|organizations|126455|
40
|EuropePMC XML fulltext|2071118|
41
|PDF fulltext|3919705|
42 56 Marek Horst
43
Second table provides numbers related to inferences generated by IIS:
44
45
|*inference type*|*count*|
46 80 Marek Horst
|dataset references|1697855|
47
|project references|1066503|
48
|software urls references|40451|
49
|protein db references|266683|
50
|research initiatives references|42709|
51
|documents classified|3165468|
52
|similar documents found|233087235|
53
|citations matched by reference text|16365015|
54
|citations matched by id|6259352|
55 56 Marek Horst
56 80 Marek Horst
Both tables are based on IIS report generated on May 9, 2016 for OpenAIRE production infrastructure (citation matching outcome numbers are related to report generated on November 19 2017, software urls references count was taken from report generated on March 21, 2018).
57 56 Marek Horst
58
h3. Funders
59
60 60 Marek Horst
Table below presents number of publications funded by different funders:
61 56 Marek Horst
62 60 Marek Horst
|*funder/funding*|*count*|
63
|EC - FP7|195151|
64
|EC - H2020|1782|
65
|FCT|23849|
66
|WT|16108|
67
|NSF (US)|119550|
68
|NHMRC (AU)|5785|
69
|ARC (AU)|8053|
70 56 Marek Horst
71 71 Marek Horst
Presented numbers are based on production infrastructure state as of December 14, 2016.
72 56 Marek Horst
73 47 Jochen Schirrwagen
h3. Data Sources
74 48 Jochen Schirrwagen
75 1 Marek Horst
Data sources in OpenAIRE are divided in: 
76 71 Marek Horst
* data providers (~805) OpenAIRE continuously collects information from. Need to comply with the "OpenAIRE Guidelines":https://guidelines.openaire.eu
77 57 Jochen Schirrwagen
* data sources (~6562) where metadata are collected from compatible data providers (usually aggregators) and represented in the OpenAIRE portal as links to the publications and datasets (_downloadFrom_).
78 1 Marek Horst
79 51 Jochen Schirrwagen
The table below presents number of data sources grouped by OpenAIRE compatibility level and data source type:
80 52 Marek Horst
81 49 Jochen Schirrwagen
|*data source type*|*compatibility*|*count*|
82 20 Jochen Schirrwagen
|institutional repository |openaire-3.0 |122 |
83 18 Alessia Bardi
| |openaire-2.0+ |103 |
84 55 Marek Horst
| |openaire-2.0 |10 |
85
| |openaire-basic |252 |
86 45 Jochen Schirrwagen
|thematic repository |openaire-3.0 |5 |
87
| |openaire-2.0+ |9 |
88 1 Marek Horst
| |openaire-basic |22 |
89
|other publication repository |openaire-3.0 |8 |
90 18 Alessia Bardi
| |openaire-2.0+ |1 |
91 20 Jochen Schirrwagen
| |openaire-2.0 |2 |
92
| |openaire-basic |34 |
93 1 Marek Horst
|data repository |openaire-data-2.0 |11 |
94 55 Marek Horst
|publication repository aggregator |openaire-3.0 |5 |
95
| |openaire-2.0 |1 |
96 29 Marek Horst
| |openaire-basic |7 |
97
|data repository aggregator |openaire-data |1 |
98
|journals |openaire-3.0 |71 |
99
| |openaire-2.0 |2 |
100
| |openaire-basic |77 |
101
|journal aggregator / publisher |openaire-3.0 |4 |
102
| |openaire-basic |26 |
103
| |proprietary |1 |
104
|publication catalogue |proprietary |12 |
105
|repository registries|proprietary|2 |
106 1 Marek Horst
|funder databases |proprietary |8 |
107
108 30 Marek Horst
The number of data sources is based on the OpenAIRE-index from December 15, 2016.
109
110
h3. Zenodo Content Status
111
112
|*data type*|*count or size*|
113 77 Marek Horst
|records|378k|
114
|managed files|1.2M|
115
|files total size|38TB|
116 30 Marek Horst
117 77 Marek Horst
Presented numbers are based on Zenodo repository status on March 2018.
118 18 Alessia Bardi
119
h2. [[OpenAIRE workflows]]
120
121
The OpenAIRE aggregation system is based on the "D-NET software toolkit":http://www.d-net.research-infrastructures.eu/. D-NET is a service-oriented framework specifically designed to support developers at constructing custom aggregative infrastructures in a cost-effective way. D-NET offers data management services capable of providing access to different kinds of external data sources, storing and processing information objects of any data models, converting them into common formats, and exposing information objects to third-party applications through a number of standard access API. Most importantly, D-NET offers infrastructure enabling services that facilitate the construction of domain-specific aggregative infrastructures by selecting and configuring the needed services and easily combining them to form autonomic data processing workflows. 
122
123
The Enabling Layer contains the Services supporting the application framework. These provide functionalities such as Service registration, discovery, subscription and notification and data transfer mechanisms through ResultSet Services. Most importantly, these Services are configured to orchestrate Services of other layers to fulfil the OpenAIRE specific requirements and implement the *[[OpenAIRE workflows]]*.
124
125
h2. Services
126
127
h3. Software life-cycle
128
129
h4. D-NET services
130 1 Marek Horst
131 73 Claudio Atzori
The D-NET services are shipped as web applications and deployed on the tomcat application server (v7.0.52) on three distinct systems: dev, beta, production. To support the deployment process all the software codebase is versioned on "SVN":https://svn-public.driver.research-infrastructures.eu/driver/dnet45 and the project is managed with maven, whose artifacts are automatically built on a continuous integration system ("Jenkins":https://jenkins-dnet.d4science.org) and hosted on a dedicated maven repository ("nexus":http://maven.research-infrastructures.eu/nexus), while webapp builds are made available via "http server":http://ppa.research-infrastructures.eu/ci_upload. The mentioned tools supporting the software lifecycle are maintained by CNR.
132 1 Marek Horst
133
The D-NET services deployment is performed in subsequent stages:
134
* The development infrastructure plays the role of test bench for the integration of the software developed by different institutions. It is maintained by CNR and runs mostly un-released code and contains mock or subsets of the data available on the production system.
135
* The beta infrastructure runs only released code. It is maintained by ICM and consists of the final integration stage where all the system workflows are tested on the real data (not necessarily the same data as the production system) before making them available to the production system. Although the software running on the beta system is not yet production ready, its portal is publicly accessible in order to showcase new features and data.
136
* The production infrastructure is maintained by ICM and runs only code that was tested on the beta system.
137 18 Alessia Bardi
138 1 Marek Horst
D-NET backend services are packed in four different web applications, each of them running on a dedicated tomcat instance.
139 18 Alessia Bardi
140 1 Marek Horst
h4. Information Inference Service
141 26 Marek Horst
142 59 Marek Horst
"Information Inference Service":https://github.com/openaire/iis is an oozie application providing set of inference algorithms deployed on Hadoop cluster. Each new IIS release is deployed into unique, dedicated HDFS location. This way different IIS versions can be utilized by different D-Net environments: beta and production. 
143 1 Marek Horst
144 59 Marek Horst
Formerly IIS was being deployed on CDH4 cluster. Since October 1st, 2015 dedicated "cdh5 branch":https://github.com/openaire/iis/commits/cdh5 was created where new SPARK modules were introduced and existing modules were optimized. On November 20, 2016 for the first time all inferences in production infrastructure were generated by IIS deployed on new CDH5 OCEAN cluster. Both stability and major performance increase were noticed.
145
146 63 Marek Horst
Currently IIS heavily depends on the following technologies:
147 68 Marek Horst
* "Java Platform Standard Edition (Java SE) 8":http://docs.oracle.com/javase/8/docs/
148 63 Marek Horst
* "Apache Avro":https://avro.apache.org/ hdfs data serialization
149
* "Apache Hadoop":http://hadoop.apache.org/ framework for distributed computing
150 64 Marek Horst
* "Apache Oozie":http://oozie.apache.org/ workflow definitions
151 63 Marek Horst
* "Apache Pig":https://pig.apache.org transformers, documents similarity
152
* "Apache Spark":http://spark.apache.org affiliation and citation matching algorithms
153 72 Marek Horst
* "Kryo":https://github.com/EsotericSoftware/kryo serialization framework
154 63 Marek Horst
* "Cermine":https://github.com/CeON/CERMINE metadata and text extraction from PDF documents
155
* "Madis":https://github.com/madgik/madis documents classification and reference extraction algorithms
156
157 1 Marek Horst
"Continuous Integration System":http://ci.ceon.pl/view/IIS/ hosted @ICM allows both building IIS artifacts and running full suite of integration tests which are triggered nightly.
158 59 Marek Horst
159 63 Marek Horst
More details on service versioning and deployment can be found on "IIS versioning and deployment":https://issue.openaire.research-infrastructures.eu/projects/openaire/wiki/IIS_versioning_and_deployment wiki page.
160 18 Alessia Bardi
161 1 Marek Horst
h4. Portal
162 23 Marek Horst
163
OpenAIRE portal is hosted at ICM. It uses Joomla! 3.6.2, a free dynamic portal engine and content management system (CMS).
164
165
The Joomla! depends on other upstream applications:
166
167
* Apache 2.4.7
168
* PHP 5.5.9
169
* MySQL 5.5.53
170 1 Marek Horst
* OpenLDAP 2.4.31
171 18 Alessia Bardi
172 1 Marek Horst
h4. Zenodo
173
174
Zenodo repository employs an instance of the Invenio software, developed by CERN.
175 31 Marek Horst
176
Repository is deployed in a production system (https://zenodo.org) and a QA system (https://sandbox.zenodo.org). In total the two systems are running on 30 VMs hosted in CERNs OpenStack infrastructure. All machines are configured using Puppet and are running on top of CERN CentOS 7.2.
177
178
Zenodo/Invenio depends on the following applications:
179
- HAProxy for load balancing
180
- Nginx for serving static content and proxying request to application server
181
- UWSGI as application server for Zenodo/Invenio application
182
- Redis for memory cache
183
- RabbitMQ as message broker
184
- Celery as distributed task queue
185
- PostgreSQL as database
186
- Elasticsearch as search engine
187
188 1 Marek Horst
Deployment process is described at http://zenodo.readthedocs.io/projectlifecycle.html#release-process
189 33 Marek Horst
190
See https://github.com/zenodo/zenodo/commits/production for changes to Zenodo production system (does not include changes to Invenio modules).
191 35 Marek Horst
192
Zenodo repository was relaunched on Invenio v3 alpha on September 12, 2016.
193 18 Alessia Bardi
194 1 Marek Horst
h3. Infrastructure services
195 18 Alessia Bardi
196 1 Marek Horst
Because OpenAIRE2020 services are a continuation and incremental extension of the services already present that resulted from OpenAIRE+ project, so they are still hosted on that same machines. More details are available at "OpenAIRE+ WP5 Maintenance Report":http://wiki.openaire.eu/xwiki/bin/view/OpenAIREplus%20Specific/WP5%20Maintenance%20Report.
197 18 Alessia Bardi
198 1 Marek Horst
h4. Hadoop clusters
199 18 Alessia Bardi
200 1 Marek Horst
h5. DM hadoop cluster
201
202
CDH version: @cdh4.3.1@
203
204
h5. IIS hadoop cluster
205 24 Marek Horst
206
There were two IIS clusters deployed:
207 39 Marek Horst
* old CDH4 IIS cluster, version @cdh4.3.1@, in operation until December 9, 2016 
208 1 Marek Horst
* new CDH5 IIS cluster deployed on March 22, 2016 in OCEAN infrastructure, supports MRv2 on YARN and SPARK
209 24 Marek Horst
210
CDH5 cluster version history:
211 27 Marek Horst
** @5.5.2@ deployment on March 22, 2016
212
** @5.5.2 -> 5.7.5@ upgrade on November 30, 2016
213 24 Marek Horst
** @5.7.5 -> 5.9.0@ upgrade on December 8, 2016
214 1 Marek Horst
215
h4. Databases
216 28 Marek Horst
217
|_database type_|_usage_|_version_|
218 24 Marek Horst
| postgress | statistics | @9.1.23@ |
219
| postgress | DNet services | @9.3.14@ |
220
| mongodb | DNet services | @3.2.6@ |
221 1 Marek Horst
| virtuoso | LOD | @7.2@ |
222 41 Marek Horst
223 18 Alessia Bardi
h4. Piwik analytics platform
224 24 Marek Horst
225
Currently deployed Piwik version: @2.17.1@, since December 6, 2016.
226 40 Marek Horst
227 24 Marek Horst
h4. ownCloud filesync platform
228 40 Marek Horst
229 18 Alessia Bardi
Deployed at https://box.openaire.eu. Current version: @8.2.7@, since October 19, 2016.
230 1 Marek Horst
231 76 Marek Horst
h3. Services monitoring
232
233
The following technologies are in use at ICM for monitoring purposes:
234
* *Zabbix* - enterprise-class open source monitoring solution for networks, servers and applications
235
* *Monit* - open source tool for monitoring and managing, processes, files, directories and file systems
236
* *Pingdom* - website performance and availability monitoring
237
* *Uptime* - website availability monitoring from worldwide locations
238
239
Their main purpose is to cover the following group of monitoring activities:
240
* Availability
241
** Zabbix - monitoring local network health
242
** pingdom, uptime - services probing
243
* Watchdogs
244
** Monit
245
** Zabbix
246
Both are used for a large variety of monitoring aspects such as:
247
** disk space
248
** cpu usage
249
** network bandwidth usage
250
** services monitoring
251
** automatic services restarting
252
among many others.
253
* Alerts
254
** Monit
255
** Zabbix
256
Both having appropriate thresholds set and both sending emails when those thresholds are exceeded.
257
258
All those monitoring technologies are in use for all production machines and services monitoring. The most crucial machines and services from BETA infrastructure are also being monitored.
259
260
h4. Hadoop cluster monitoring
261
262
CDH5 IIS hadoop cluster was configured and set up using Cloudera Manager instance deployed at ICM OCEAN data center. CM provides many features for monitoring health and performance of the cluster components (hosts and service daemons) as well as the performance and resource demands of the jobs running on cluster. Apart from regular monitoring of various characteristics of the whole system (e.g. CPUs utilization, network/disk/hdfs IO) it is also capable of sending alerts whenever any host on a cluster goes down or when resources are severely exhausted. 
263
264 22 Marek Horst
h3. Architectural changes
265 1 Marek Horst
266 58 Marek Horst
Detailed list of all administrative operations undertaken @ICM is available on [[D64_Servers_Administration_Operations_Changelog|changelog for servers administration operations]] wiki page.
267 18 Alessia Bardi
268 62 Marek Horst
h4. Introducing CDH5 IIS cluster hosted in ICM OCEAN infrastructure
269 1 Marek Horst
270
Slave node specification:
271
* Huawei RH1288 V3
272
* CPU: 2x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (24 cores, 48 threads)
273
* RAM: 128GB
274
* HDD: 4x SATA 6TB 7.2K RPM (HDFS)
275
276
Cluster summary (16 slaves):
277
* CPU: 384 cores, 768 threads
278
* RAM: 2048GB
279
* HDD: 384TB (HDFS)
280
281
YARN available resources:
282
* vcores: 640
283
* memory: 1.44TB
284 18 Alessia Bardi
* HDFS: 344TB
285 1 Marek Horst
286
h4. Incorporating resources from old CDH4 IIS cluster into existing DM CDH4 cluster
287 25 Marek Horst
288 1 Marek Horst
This task was possible after shutting down old IIS CDH4 cluster what happened on December 9, 2016.
289 25 Marek Horst
290
h4. Deploying DNet postgress and mongodb databases on separate machines
291
292
Separating @openaire-services@ database instances into dedicated ones (since June 27, 2016):
293
* @openaire-services-postgresql@
294
* @openaire-services-mongodb@
295 34 Marek Horst
296 1 Marek Horst
h4. Updating Zenodo repository infrastructure at CERN
297 34 Marek Horst
298
Several architectural changes were introduced in CERN's infrastructure:
299
* changed storage backend from OpenAFS to CERN EOS (18 PB disk cluster) for better scalability
300
* changed from self-managed MySQL database to CERN Database Team managed PostgreSQL database
301
* deployment of Elasticsearch clusters (6 VMs)
302
* SLC to CentOS 7 on all 30 VMs
303 1 Marek Horst
304
h3. System downtimes
305 36 Marek Horst
306
h4. @ICM
307 1 Marek Horst
308 75 Marek Horst
* [unplanned] November 13, 2017, 8 hours. #3284 main openaire.eu site becoming unavailable, two hard drives down one after another in a 8h period overnight
309 36 Marek Horst
* [planned] November 14, 2016, 2 hours. #2423 dealing with Linux Dirty COW vulnerability: kernel upgrade, OpenAIRE services restart.
310
311
h4. @CERN
312
313
* [unplanned] September 16, 2016, 3 hours. Preparation of a new load balancer caused an automatic update of CERN outerperimeter firewall that automatically closed access to the operational load balancers.
314
* [planned] September 12, 2016, 8 hours. Complete migration from old infrastructure to new infrastructure
315 1 Marek Horst
* minor incidents until September 12, 2016 due to overload of the legacy system