OpenAIRE2020 - Wiki

h1. Aggregation subsystem

OpenAIRE infrastructure collects objects from a set of data sources of different typologies, e.g. repositories, CRISs, dataset archives, aggregators, entity registries, journals. All these contain information relative to different interrelated objects of the OpenAIRE data model and expose them via a specific protocol and exchange format. In general, data sources expose *information packages* (i.e. metadata records) describing a *primary object* and possibly other *sub-objects*. For example, an OpenAIRE compliant repository delivers information packages (i.e., metadata records) which contain information about a publication result object, but also the (author) persons who created such result, the projects funding such result, etc. The aggregation system is in charge of collecting heterogeneous information packages and transforming them onto internal homogeneous formats, which the data provision subsystem can directly transform onto the *native information space graph*. To this aim, the aggregation system has to cover the following responsibilities:

* managing an arbitrary large set of data sources;

* define, monitor, and execute ingestion workflows for all data sources, based on their typology (e.g. repository, journals, aggregators), protocol (e.g. OAI-PMH, FTP, JDBC, local file system), exchange format (e.g. XML, CSV, JSON), primary entity type (e.g. publication, dataset, projects).

* implement specific *stateless identifier* policies: guaranteeing that if the same objects or relationships are collected more than once from the same data sources, they will be assigned the same identifier in OpenAIRE.

p=. !{width:45%}informationPackages_Objects.png!

Figure 1 – Information packages and mapping onto native objects

The following sections will introduce the notions of:

* Information packages and ingestion workflows;

* How to assign a stateless (and permanent) identifiers to the primary objects when they enter the information space;

For each of these aspects, an explanation of the problem, an extension of the data model to handle the problem, and, where necessary, a solution to the problem using the updated model are provided. 

h2. Information packages

In OpenAIRE objects are collected from external data sources in the form of *information packages*. In particular, an information package is a file in some interpretable format (e.g. XML, JSON), which contains a *data source-assigned identifier* (mandatory) and information (e.g., properties) relative to a primary object, of a given object type (entity). In some cases, an information package may be relative to a set of primary objects, as in the case of CSV files. Beyond the primary object, an information package may contain information (but not necessarily the identifier) relative to other objects (of likely different types), called sub-objects, which must be directly or indirectly associated with the package primary object. Figure 1 shows an example of an information package whose primary object is 1: for example, an information package from OpenDOAR is relative to a repository data source object and can be identified by the relative OpenDOAR identifier. Its sub-objects are those from 2 to 6: for example, an OpenDOAR package also contains information about the organization object responsible for the repository data source.

h2. Ingestion workflows 

We call ingestion workflow the process that takes the information package from a data source and transforms them onto OpenAIRE entity formats. A workflow is therefore dependent on:

* The data source typology;

* The access method, namely (i) protocol required to get the data (e.g., OAI-PMH, JDBC, FTP) and (ii) relative access configuration (e.g., entry point, parameters, etc.);

* The primary object type of the information packages;

* The exchange format of the information package at hand. 

Note that data sources of the same typology may deliver information packages relative to different primary entity types (in general we can assume they will do it from different access points). For example, CRIS systems may expose through OAI-PMH both publication or project primary objects. To be sustainable, the OpenAIRE infrastructure requires data sources of a given typology to comply to specific export guidelines. 

*Information package structure (OpenAIRE guidelines)* The OpenAIRE infrastructure will includes services capable of handling automated collection of information packages from data sources according to given population workflows. To this aim, the OpenAIRE guidelines will describe which XML information packages structure should be expected for each population workflow triple:

<pre>

<datasource typology, access method, primary entity type> → XML information package structure

</pre>

available to the system. WP6 will develop services to automatically process the information packages and insert the relative entities onto the information space.

Information package heterogeneity and harmoniztion Unfortunately, the “raw” information packages exported by data sources will likely not match the information package structures to be identified in the previous step. For example, CRIS systems generally support OAI-PMH harvesting of information, but may export information packages relative to the same entities (e.g., projects, publications) in different XML formats. To this aim, WP6 will update its transformation services in order to map the specific structures exposed by a data source through a given workflow so that they match the expected information package structure.

p=. !{width:45%}informationPackages_Ingestion.png!

Figure 2 – Information Packages: ingestion workflows (AM = Access Method, DS = Data source typology, PET = Primary Entity Type, F = information package Format): data sources of the same typology export the same primary entity of the same type through different “raw” information package format structures.

h2. Identity of original entities

Original entities reach the information space from different workflows. Once they enter the information space they must be assigned a unique “stateless” identifier. The data sources of such entities are not under the OpenAIRE infrastructure control and may in any moment decide to delete, update, or add new entities or relationships between them. Hence, it is particularly important to make sure such identifiers are generated from the incoming information packages in a stateless and stable way that is “if the same entity enters the information space at different times, it will be assigned the same identifier”. 

To this aim, the OpenAIRE infrastructure constructs indentifiers for primary entities and sub-entities in an information package by combining three levels of scope: data sources, relative primary entities, and sub-entities of such primary entities. More specifically:

* Infrastructure scope: all data sources are registered and assigned a unique identifier in OpenAIRE; 

* Data source scope: information packages from the same data source contain one primary entity with an identifier which is unique in the context of the data source;

* Primary entity scope: information packages may contain a number of sub-entities relative to the primary entity; unlike primary entities, sub-entities may not necessarily come with an identifier (data source scope)  and can be generally uniquely identified in the scope of the primary entity based on their properties. The process of identification of such “unique information” is very much dependent on the given information package structure.

Primary entity identifiers The process of generation of stateless identifiers for primary entities is based on a data source scope strategy. Independently of the workflows, the type of entity, and the data source kind, primary entity identifiers are always obtained by concatenating the name space prefix of the data source with the primary entity identifier (using and underscore): 

nameSpacePrefix_mainEntityID

Although one may consider an infrastructure scope strategy, where the assumption is that all primary entities identifers are persistent identifiers, therefore unique across several data sources, in OpenAIRE this is not generally the case, hence we adopt a common and safe strategy of identifier generation.

Sub-entity identifiers Assigning identifiers to sub-entities can be performed following different strategies, some more “optimistic” and some more “pessimistic” about the ability of inferring unambiguous “unique information” for sub-entities from their properties in the information package. For example, one may assume that:

* Infrastructure scope strategy: sub-entities with the same “unique information” are collapsed in the same entity across different data sources (infrastructure scope). Entity splitting will identify and solve possible entity “overloads” in a second stage. 

<pre>

uniqueInformation

</pre>

* Data source scope strategy: Sub-entities with the same “unique information” are collapsed in the same entity but only within the same data source scope. Entity splitting will identify and solve possible entity “overloads” in a second stage.

<pre>

nameSpacePrefix_uniqueInformation

</pre>

* Primary entity scope strategy: Sub-entities with the same “unique information” are collapsed in the same entity but only within the same primary entity scope (see Figure 9). De-duplication of entities will solve redundancy in a second stage.

<pre>

nameSpacePrefix_mainEntityID_uniqueInformation

</pre>

Assigning identifiers to sub-entities follows different strategies depending on the specific workflow, hence the relative information packages structure.

p=. !{width:45%}identifiers_primaryObjectScope.png!

Figure 3 – Assigning unique identifiers to sub-entities: primary entity scope

p=. !{width:45%}identifiers_dataSourceScope.png!

Figure 4 – Assigning unique identifiers to sub-entities: data source scope