Aggregation subsystem » History » Revision 8
Revision 7 (Paolo Manghi, 13/05/2015 04:18 PM) → Revision 8/22 (Paolo Manghi, 13/05/2015 04:32 PM)
h1. Aggregation subsystem
OpenAIRE infrastructure collects objects from a set of data sources of different typologies, e.g. repositories, CRISs, dataset archives, aggregators, entity registries, journals. All these contain information relative to different interrelated objects of the OpenAIRE data model and expose them via a specific protocol and exchange format. In general, data sources expose *information packages* (i.e. metadata records) describing a *primary object* and possibly other *sub-objects*. For example, an OpenAIRE compliant repository delivers information packages (i.e., metadata records) which contain information about a publication result object, but also the (author) persons who created such result, the projects funding such result, etc. The aggregation system is in charge of collecting heterogeneous information packages and transforming them onto internal homogeneous formats, which the data provision subsystem can directly transform onto the *native information space graph*. To this aim, the aggregation system has to cover the following responsibilities:
* managing an arbitrary large set of data sources;
* define, monitor, and execute ingestion workflows for all data sources, based on their typology (e.g. repository, journals, aggregators), protocol (e.g. OAI-PMH, FTP, JDBC, local file system), exchange format (e.g. XML, CSV, JSON), primary entity type (e.g. publication, dataset, projects).
* implement specific *stateless identifier* policies: guaranteeing that if the same objects or relationships are collected more than once from the same data sources, they will be assigned the same identifier in OpenAIRE.
p=. !{width:45%}informationPackages_Objects.png!
Figure 1 – Information packages and mapping onto native objects
The following sections will introduce the notions of:
* Information packages and ingestion workflows;
* How to assign a stateless (and permanent) identifiers to the primary objects when they enter the information space;
For each of these aspects, an explanation of the problem, an extension of the data model to handle the problem, and, where necessary, a solution to the problem using the updated model are provided.
h2. Information packages
In OpenAIRE objects are collected from external data sources in the form of *information packages*. In particular, an information package is a file in some interpretable format (e.g. XML, JSON), which contains a *data source-assigned identifier* (mandatory) and information (e.g., properties) relative to a primary object, of a given object type (entity). In some cases, an information package may be relative to a set of primary objects, as in the case of CSV files. Beyond the primary object, an information package may contain information (but not necessarily the identifier) relative to other objects (of likely different types), called sub-objects, which must be directly or indirectly associated with the package primary object. Figure 1 shows an example of an information package whose primary object is 1: for example, an information package from OpenDOAR is relative to a repository data source object and can be identified by the relative OpenDOAR identifier. Its sub-objects are those from 2 to 6: for example, an OpenDOAR package also contains information about the organization object responsible for the repository data source.
h2. Ingestion workflows
We call ingestion workflow the process that takes the information package from a data source and transforms them onto internal OpenAIRE entity formats. OpenAIRE internal formats are of three main kinds: OAF for publications, DFM for datasets, and PFM for projects. These are currently the only primary object types being harvested from data sources, other internal formats will be defined if necessary.
In general a A workflow is therefore dependent on:
* The data source typology;
* The access method, namely (i) protocol required to get the data (e.g., OAI-PMH, JDBC, FTP) and (ii) relative access configuration (e.g., entry point, parameters, etc.);
* The primary object type of the information packages, which implies an internal OpenAIRE format; packages;
* The exchange format of the information package at hand, which implies a mapping that maps the structure and semantics of information packages onto the corresponding OpenAIRE internal format. hand.
To be sustainable, the OpenAIRE infrastructure requires data sources of a given typology to comply to specific export "OpenAIRE guidelines":http://guidelines.openaire.eu. Guidelines are available for publication repositories (also apply to aggregators of repositories, journals, and aggregator of journals), data repositories (also apply to aggregators of data repositories), and CRIS systems. Respectively, such data sources must export information about publications, datasets, and publication/datasets/persons/projects respectively. Suggested protocols are OAI-PMH and FTP, but different (HTTP-proprietary) protocols can also be managed. Guidelines are extremely important, since assuming a uniform incoming format for given primary objects moves the complexity of ingestion workflows at the semantic level of the mappings. For other typologies of data sources, namely *entity registries*, specific workflows must be defined, as data sources changes considerably in the way they export content. In such cases the workflows will also define structural mapping from a data source information package format to the internal OpenAIRE format.
Note that data sources of the same typology may deliver information packages relative to different primary entity types and this (in general we can assume they will happen do it from different data source access points (in data source profiles, these are called APIs). points). For example, CRIS systems may expose through OAI-PMH both publication or project primary objects, and in such cases they will objects. To be associated two distinct ingestion workflows.
sustainable, the OpenAIRE infrastructure requires data sources of a given typology to comply to specific export guidelines.
*Information package structure (OpenAIRE guidelines)* The OpenAIRE infrastructure will includes services capable of handling automated collection of information packages from data sources according to given population workflows. To this aim, the OpenAIRE guidelines will describe which XML information packages structure should be expected for each population workflow triple:
<pre>
<datasource typology, access method, primary entity type> → XML information package structure
</pre>
available to the system. WP6 will develop services to automatically process the information packages and insert the relative entities onto the information space.
Information package heterogeneity and harmoniztion Unfortunately, the “raw” information packages exported by data sources will likely not match the information package structures to be identified in the previous step. For example, CRIS systems generally support OAI-PMH harvesting of information, but may export information packages relative to the same entities (e.g., projects, publications) in different XML formats. To this aim, WP6 will update its transformation services in order to map the specific structures exposed by a data source through a given workflow so that they match the expected information package structure.
p=. !{width:45%}informationPackages_Ingestion.png!
Figure 2 – Information Packages: ingestion workflows (AM = Access Method, DS = Data source typology, PET = Primary Entity Type, F = information package Format): data sources of the same typology export the same primary entity of the same type through different “raw” information package format structures.
h2. Identity of original entities
Original entities reach the information space from different workflows. Once they enter the information space they must be assigned a unique “stateless” identifier. The data sources of such entities are not under the OpenAIRE infrastructure control and may in any moment decide to delete, update, or add new entities or relationships between them. Hence, it is particularly important to make sure such identifiers are generated from the incoming information packages in a stateless and stable way that is “if the same entity enters the information space at different times, it will be assigned the same identifier”.
To this aim, the OpenAIRE infrastructure constructs indentifiers for primary entities and sub-entities in an information package by combining three levels of scope: data sources, relative primary entities, and sub-entities of such primary entities. More specifically:
* Infrastructure scope: all data sources are registered and assigned a unique identifier in OpenAIRE;
* Data source scope: information packages from the same data source contain one primary entity with an identifier which is unique in the context of the data source;
* Primary entity scope: information packages may contain a number of sub-entities relative to the primary entity; unlike primary entities, sub-entities may not necessarily come with an identifier (data source scope) and can be generally uniquely identified in the scope of the primary entity based on their properties. The process of identification of such “unique information” is very much dependent on the given information package structure.
Primary entity identifiers The process of generation of stateless identifiers for primary entities is based on a data source scope strategy. Independently of the workflows, the type of entity, and the data source kind, primary entity identifiers are always obtained by concatenating the name space prefix of the data source with the primary entity identifier (using and underscore):
nameSpacePrefix_mainEntityID
Although one may consider an infrastructure scope strategy, where the assumption is that all primary entities identifers are persistent identifiers, therefore unique across several data sources, in OpenAIRE this is not generally the case, hence we adopt a common and safe strategy of identifier generation.
Sub-entity identifiers Assigning identifiers to sub-entities can be performed following different strategies, some more “optimistic” and some more “pessimistic” about the ability of inferring unambiguous “unique information” for sub-entities from their properties in the information package. For example, one may assume that:
* Infrastructure scope strategy: sub-entities with the same “unique information” are collapsed in the same entity across different data sources (infrastructure scope). Entity splitting will identify and solve possible entity “overloads” in a second stage.
<pre>
uniqueInformation
</pre>
* Data source scope strategy: Sub-entities with the same “unique information” are collapsed in the same entity but only within the same data source scope. Entity splitting will identify and solve possible entity “overloads” in a second stage.
<pre>
nameSpacePrefix_uniqueInformation
</pre>
* Primary entity scope strategy: Sub-entities with the same “unique information” are collapsed in the same entity but only within the same primary entity scope (see Figure 9). De-duplication of entities will solve redundancy in a second stage.
<pre>
nameSpacePrefix_mainEntityID_uniqueInformation
</pre>
Assigning identifiers to sub-entities follows different strategies depending on the specific workflow, hence the relative information packages structure.
p=. !{width:45%}identifiers_primaryObjectScope.png!
Figure 3 – Assigning unique identifiers to sub-entities: primary entity scope
p=. !{width:45%}identifiers_dataSourceScope.png!
Figure 4 – Assigning unique identifiers to sub-entities: data source scope