D64 OpenAIRE Maintenance Report » History » Revision 22
Revision 21 (Lars Holm Nielsen, 14/12/2016 12:58 PM) → Revision 22/83 (Marek Horst, 14/12/2016 05:32 PM)
h1. D6.4 OpenAIRE Maintenance Report (v1, 9th of December 2016)
{{toc}}
h2. Overview
This document contains information about the deployment, status of OpenAIRE2020 services and content, and the history of major modifications of the system hosted at ICM, Poland.
The official maintenance of the OpenAIRE2020 services began on January 1st, 2015 when the project started.
TODO: elaborate the following
The deliverable will consist in a high-level report on the status of
* OpenAIRE workflows (CNR),
* services (ICM),
* and Information Space (UNIBI).
h2. Information Space
Brief description of the data model and status of content, i.e. numbers about data providers, their typology, publications, datasets, links... etc.
@UNIBI: your contribution is needed here
h3. Data Provider
h3. Content Status
|*data type*|*count*|
|publications|17460368|
|datasets|3226586|
|projects|653268|
|organizations|64591|
|authors|16188328|
|EuropePMC XML fulltext|1574358|
|PDF fulltext|2227458|
Table based on IIS report generated on November 20, 2016.
h2. [[OpenAIRE workflows]]
The OpenAIRE aggregation system is based on the "D-NET software toolkit":http://www.d-net.research-infrastructures.eu/. D-NET is a service-oriented framework specifically designed to support developers at constructing custom aggregative infrastructures in a cost-effective way. D-NET offers data management services capable of providing access to different kinds of external data sources, storing and processing information objects of any data models, converting them into common formats, and exposing information objects to third-party applications through a number of standard access API. Most importantly, D-NET offers infrastructure enabling services that facilitate the construction of domain-specific aggregative infrastructures by selecting and configuring the needed services and easily combining them to form autonomic data processing workflows.
The Enabling Layer contains the Services supporting the application framework. These provide functionalities such as Service registration, discovery, subscription and notification and data transfer mechanisms through ResultSet Services. Most importantly, these Services are configured to orchestrate Services of other layers to fulfil the OpenAIRE specific requirements and implement the *[[OpenAIRE workflows]]*.
h2. Services
How the system is maintained
h3. Software life-cycle
h4. D-NET services
The D-NET services are shipped as web applications and deployed on the tomcat application server (v7.0.52) on three distinct systems: dev, beta, production. To support the deployment process all the software artifacts are automatically built on a continuous integration system ("Jenkins":https://ci.research-infrastructures.eu) and hosted on a dedicated maven repository ("nexus":http://maven.research-infrastructures.eu/nexus), while webapp builds are made available via "http server":http://ppa.research-infrastructures.eu/ci_upload. The mentioned tools supporting the software lifecycle are maintained by CNR.
The D-NET services deployment is performed in subsequent stages:
* The development infrastructure plays the role of test bench for the integration of the software developed by different institutions. It is maintained by CNR and runs mostly un-released code and contains mock or subsets of the data available on the production system.
* The beta infrastructure runs only released code. It is maintained by ICM and consists of the final integration stage where all the system workflows are tested on the real data (not necessarily the same data as the production system) before making them available to the production system. Although the software running on the beta system is not yet production ready, its portal is publicly accessible in order to showcase new features and data.
* The production infrastructure is maintained by ICM and runs only code that was tested on the beta system.
D-NET backend services are packed in four different web applications, each of them running on a dedicated tomcat instance.
_@Claudio: could you fill that in? Probably you can rely on http://wiki.openaire.eu/xwiki/bin/view/OpenAIREplus%20Specific/WP5%20Maintenance%20Report_
h4. Information Inference Service
"Information Inference Service":https://github.com/openaire/iis versioning and deployment is described on "IIS versioning and deployment":https://issue.openaire.research-infrastructures.eu/projects/openaire/wiki/IIS_versioning_and_deployment wiki page.
h4. Portal
_@Paweł: could you fill that in? You can do it in a similar way it was done in http://wiki.openaire.eu/xwiki/bin/view/Technical%20Specifications/Maintenance%20Report_
h4. Zenodo
Zenodo repository is deployed according to the process described at http://zenodo.readthedocs.io/projectlifecycle.html#release-process
Zenodo repository employs an instance of the Invenio software, developed by CERN.
_@Lars: could you fill that in? You can do it in a similar way it was done in http://wiki.openaire.eu/xwiki/bin/view/Technical%20Specifications/Maintenance%20Report_
h3. Infrastructure services
Because OpenAIRE2020 services are a continuation and incremental extension of the services already present that resulted from OpenAIRE+ project, so they are still hosted on that same machines. More details are available at "OpenAIRE+ WP5 Maintenance Report":http://wiki.openaire.eu/xwiki/bin/view/OpenAIREplus%20Specific/WP5%20Maintenance%20Report.
_@Paweł: are there any new services in maintenance since OpenAIRE+ project worth mentioning here?_
h4. Hadoop clusters
h5. DM hadoop cluster
CDH version: @cdh4.3.1@
h5. IIS hadoop cluster
CDH version: @cdh4.3.1@, replaced with @cdh5.5.2@ on November 20, 2016, to be replaced with @cdh5.9.0@
h4. Databases
* postgres
* mongodb
* virtuoso
h4. Piwik
h4. box.openaire.eu (?)
h3. Architectural changes
[[D64_Servers_Administration_Operations_Changelog|Change h4. Change Log for servers administration operations]] operations
_@Paweł: could you fill that in? This could be similar to: http://wiki.openaire.eu/xwiki/bin/view/OpenAIREplus%20Specific/Change%20Log%20for%20servers%20administration%20operations_
h4. Introducing CDH5 IIS cluster hosted in OCEAN infrastructure
Slave node specification:
* Huawei RH1288 V3
* CPU: 2x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (24 cores, 48 threads)
* RAM: 128GB
* HDD: 4x SATA 6TB 7.2K RPM (HDFS)
Cluster summary (16 slaves):
* CPU: 384 cores, 768 threads
* RAM: 2048GB
* HDD: 384TB (HDFS)
YARN available resources:
* vcores: 640
* memory: 1.44TB
* HDFS: 344TB
h4. Incorporating resources from old CDH4 IIS cluster into existing DM CDH4 cluster
_@Paweł: are you aware of any other major modifications regarding other services such as new portal version etc?_
h3. System downtimes
_@Paweł: could you fill that in? We could create dedicated with entry similar to:
http://wiki.openaire.eu/xwiki/bin/view/OpenAIREplus%20Specific/History%20of%20system%20outages
and link it here._