Aggregation subsystem¶

OpenAIRE infrastructure collects objects from a set of data sources of different typologies, e.g. repositories, CRISs, dataset archives, aggregators, entity registries, journals. All these contain information relative to different interrelated objects of the OpenAIRE data model and expose them via a specific protocol and exchange format. In general, data sources expose information packages (i.e. metadata records) describing a primary object and possibly other secondary objects. For example, an OpenAIRE compliant repository delivers information packages (i.e., metadata records) which contain information about a publication result object, but also the (author) persons who created such result, the projects funding such result, etc. The aggregation system is in charge of collecting heterogeneous information packages and transforming them onto internal homogeneous formats, which the data provision subsystem can directly transform onto the native information space graph. To this aim, the aggregation system has to cover the following responsibilities:

• managing an arbitrary large set of data sources;
• define, monitor, and execute ingestion workflows for all data sources, based on their typology (e.g. repository, journals, aggregators), protocol (e.g. OAI-PMH, FTP, JDBC, local file system), exchange format (e.g. XML, CSV, JSON), primary entity type (e.g. publication, dataset, projects).
• implement specific stateless identifier policies: guaranteeing that if the same objects or relationships are collected more than once from the same data sources, they will be assigned the same identifier in OpenAIRE.

Figure 1 – Information packages and mapping onto native objects

The following sections will introduce the notions of:

• Information packages and population workflows: entity ingestion from data sources into the information space;
• How to assign a stateless (and permanent) identifiers to the entities when they enter the information space;

For each of these aspects, an explanation of the problem, an extension of the data model to handle the problem, and, where necessary, a solution to the problem using the updated model are provided.

Information packages¶

In OpenAIRE entities are collected from external data sources in the form of “information packages”. This notion aims at generalizing the OpenAIRE scenario of bibliographic metadata records import from repository data sources to other data source typologies and other types of (primary) entities. In particular, we shall call information package a file in some interpretable format (e.g. XML), which contains identifier and information (e.g., properties) relative to one entity, called primary entity, of a given entity type. An information package may contain information (but not necessarily the identifier) relative to other entities (of likely different entity types), called sub-entities, which must be directly or indirectly associated with the package primary entity. Figure 7 shows an example of an information package whose primary entity is 1: for example, an information package from OpenDOAR is relative to a repository data source and can be identified by the relative OpenDOAR identifier. Its sub-entities are those from 2 to 6: for example, an OpenDOAR package also contains information about the organization responsible for the repository data source.

Population Workflows¶

Original entities are collected from information packages originating from various data sources. We call population workflow the process that takes the information package from a data source, extracts its primary entity and its related entities, and stores them into the OpenAIRE information space. A workflow is therefore dependent on:

• The data source typology (including the expert-validate entity pool);
• The access method, namely (i) protocol required to get the data (e.g., OAI-PMH, JDBC, FTP) and (ii) relative access configuration (e.g., entry point, parameters, etc.);
• The primary entity type of the information packages;
• The XML structure of the information packages at hand, which depends on the primary entity type.

Note that data sources of the same typology may deliver information packages relative to different primary entity types (in general we can assume they will do it from different access points). For example, CRIS systems may expose through OAI-PMH both publication or project primary entities.
Information package structure (OpenAIRE guidelines) The OpenAIRE infrastructure will includes services capable of handling automated collection of entities from data sources according to given population workflows. To this aim, the OpenAIRE guidelines will describe which XML information packages structure should be expected for each population workflow triple
<datasource typology, access method, primary entity type> → XML information package structure
available to the system. WP6 will develop services to automatically process the information packages and insert the relative entities onto the information space.
Information package heterogeneity and harmoniztion Unfortunately, the “raw” information packages exported by data sources will likely not match the information package structures to be identified in the previous step. For example, CRIS systems generally support OAI-PMH harvesting of information, but may export information packages relative to the same entities (e.g., projects, publications) in different XML formats. To this aim, WP6 will update its transformation services in order to map the specific structures exposed by a data source through a given workflow so that they match the expected information package structure.

Figure 2 – Information Packages: ingestion workflows (AM = Access Method, DS = Data source typology, PET = Primary Entity Type, F = information package Format): data sources of the same typology export the same primary entity of the same type through different “raw” information package format structures.

Identity of original entities¶

Original entities reach the information space from different workflows. Once they enter the information space they must be assigned a unique “stateless” identifier. The data sources of such entities are not under the OpenAIRE infrastructure control and may in any moment decide to delete, update, or add new entities or relationships between them. Hence, it is particularly important to make sure such identifiers are generated from the incoming information packages in a stateless and stable way that is “if the same entity enters the information space at different times, it will be assigned the same identifier”.
To this aim, the OpenAIRE infrastructure constructs indentifiers for primary entities and sub-entities in an information package by combining three levels of scope: data sources, relative primary entities, and sub-entities of such primary entities. More specifically:

• Infrastructure scope: all data sources are registered and assigned a unique identifier in OpenAIRE;
• Data source scope: information packages from the same data source contain one primary entity with an identifier which is unique in the context of the data source;
• Primary entity scope: information packages may contain a number of sub-entities relative to the primary entity; unlike primary entities, sub-entities may not necessarily come with an identifier (data source scope) and can be generally uniquely identified in the scope of the primary entity based on their properties. The process of identification of such “unique information” is very much dependent on the given information package structure.

Primary entity identifiers The process of generation of stateless identifiers for primary entities is based on a data source scope strategy. Independently of the workflows, the type of entity, and the data source kind, primary entity identifiers are always obtained by concatenating the name space prefix of the data source with the primary entity identifier (using and underscore):

nameSpacePrefix_mainEntityID

Although one may consider an infrastructure scope strategy, where the assumption is that all primary entities identifers are persistent identifiers, therefore unique across several data sources, in OpenAIRE this is not generally the case, hence we adopt a common and safe strategy of identifier generation.
Sub-entity identifiers Assigning identifiers to sub-entities can be performed following different strategies, some more “optimistic” and some more “pessimistic” about the ability of inferring unambiguous “unique information” for sub-entities from their properties in the information package. For example, one may assume that:

• Infrastructure scope strategy: sub-entities with the same “unique information” are collapsed in the same entity across different data sources (infrastructure scope). Entity splitting will identify and solve possible entity “overloads” in a second stage.
  

  uniqueInformation
  

• Data source scope strategy: Sub-entities with the same “unique information” are collapsed in the same entity but only within the same data source scope. Entity splitting will identify and solve possible entity “overloads” in a second stage.
  

  nameSpacePrefix_uniqueInformation
  

• Primary entity scope strategy: Sub-entities with the same “unique information” are collapsed in the same entity but only within the same primary entity scope (see Figure 9). De-duplication of entities will solve redundancy in a second stage.
  

  nameSpacePrefix_mainEntityID_uniqueInformation
  

Assigning identifiers to sub-entities follows different strategies depending on the specific workflow, hence the relative information packages structure.

Figure 3 – Assigning unique identifiers to sub-entities: primary entity scope

Figure 4 – Assigning unique identifiers to sub-entities: data source scope