D-Net

#1308 reverting uri:oozie:distcp-action:0.2 change: version is not properly recognized by oozie 3.3.2-cdh4.3.1

#1260 enabling document to protein databank reference extraction in primary workflow, supporting 3 new parameters: active_referenceextraction_pdb, export_action_set_id_document_pdb, export_referenceextraction_pdb_url_root

#1301 introducing explicit export mode flags: active_export_to_hbase and active_export_to_json. This way both exports can be enabled or both of them can be disabled.

#1308 switching distcp namespace to uri:oozie:distcp-action:0.2

#1301 introducing common/export_to_json and utilizing this subworkflow in both primary and preprocessing workflows executing it when active_export=false which means hbase export is disabled

#1248 introducing fault subdirectory support in all workflows wrapping metadataextraction subworkflow up to the processing and primary root workflows. This should prevent fault directory from being removed when ${remove_sideproducts} flag is enabled, it will be propagated along with metadata and plaintext.

Removing usage of working_dir from Java workflow node.

#1195 removing obsolete ports docreation and datasetid from hbase mapred import, removing references to those ports in workflow.xml files, updating transformer by removing filtering by datasetid due to decisions made in #1072

#1176 defining remove_sideproducts property in workflows headers

#1172 introducing support for active_export parameter in both preprocessing and primary workflows

#1153 utilizing ${user.name} placeholder in ${workingDir} generation process, copying version.properties from oozie_app to mark execution environment with application version

#1147 introducing HTML import and HTML plaintext ingestion in main workflows: primary and preprocessing

#1133 dropping useless workfing_dir creation for java nodes

#118 fixing typos

#118 introducing website usage analysis as integral part of primary workflow

#1070 updating import_project_concepts_context_ids_csv default value to "fet-fp7,fet-h2020"

#1070 introducing support for multiple context identifiers, replacing import_project_concepts_context_id IIS input parameter with import_project_concepts_context_ids_csv

#1072 transition fix: replacing forking_skip_imported_data with export

#1072 dropping IIS feature filtering out already existing project and dataset references from IIS export

#919 enabling concepts matching for FET projects in mainworkflows: import, export, primary and preprocessing

#1022 introducing extracted document metadata collapser at importing phase.
Propagating extracted document mentadata (including PMC ingested metadata) to processing part of workflow what can be exploited by citation matching module.
Introducing citations collapser in last stage of processing phase collapsing ingested citations with matched citations.

#963 propagating dataset -> mdstore from import to exporting phase: importer produces DocumentToMDStore datasetore utilized by exporter module. Updating transformer definition to handle DocumentToMDStore instead of Identifier schema

introducing support for active_existence_filter, set to true by default. Setting this parameter to false allows processing contents not having its counterpart among metadata records retrieved from HBase. This solution was required to e.g. process ubiquity contents which were not present in HBase dump metadata.

renaming metadataextraction_excluded_ids to more appropriate metadataextraction_excluded_checksums

#913 introducing support for max file size parameter, currently checked against Content-Lenght header

input port name fix: input_citation->input_citations

#637 renaming document_extractedMetadata algorithm to more descriptive document_affiliations, propagating changes to action set identifier properties names

reenabling PMC ingestion when citationmatching flag is set

setting export_action_set_id_entity_dataset to $UNDEFINED$ by default, this should not be required because dataset reference extraction module might be deactivated. Check will be performed at dataset entity exporter module and when value is not set - exception will be raised.

#757 temporarily disabling PMC ingestion until fixing openaire identifiers building process

#568, #577 enabling proper citations export by introducing PMC citation ingestion and citation matching outcome merging and grouping for exporting purposes. Introducing union instead of collapser which should be introduced in near future.

removing output_citation_pmc port duplicate

#9059 reverting #717 change: shortening app_path for primary workflow due to the fix applied by Paweł on WF_JOBS MODIFY mysql table: canging varchar(255) to mediumtext.

#717 shortening app_path for primary workflow

fixing output port names: removing default values for citation_pmc and dataset, setting proper output_citation_pmc in both preprocessing and primary workflows

skipping PMC citations ingestion when citationmatching algorithm is not enabled

shortening transformer_export_documentto* action names to be less than 50 characters

#354 hooking up primary/main workflow with documenttodataset and documenttoproject transformers skipping export of already existing relations in HBase

#486 introducing last piece missing: text collapser in front of referenceextraction_researchinitiatives joining text contents coming from already existing document_text input port and newly introduced document_text_wos input port providing WoS contents

intregrating pmc citations ingestion with primary workflow, adjust port names, deduplicating dependencies

skipping exporting citation matching outcome