D-Net

updating job.properties

updating job.properties

#1397 removing obsolete parameters in subworkflow actions definitions

#1209 introducing support for trust level thresholds provided as IIS input parameter

removing obsolete quick run workflows

updating job.properties

#1381 porting pmc citations ingestion from cascading framework to pig. Moving code from icm-iis-ingest-pmc to icm-iis-transformers including itegration tests, removing obsolete scala code along with unneded dependencies. Switching subworkflow in primary workflow.

updating job.properties

#1339 fixing input_dedup_map in pmc citation ingestion when match_content_with_metadata=false. Should not be set dynamically but statically, it will be enabled only when metadata_import is enabled

#1339 replacing active_existence_filter flag with match_content_with_metadata and changing identifiers matching logic: when flag is disabled neither contents identifiers will be filtered nor deduplicated against metadata identifiers. Up unit now, when active_existence_filter flag was disabled contents were deduplicated which is not desired when running IIS in standalone mode on contents having their representatives in HBase

#1329 enabling pmc ingestion when active_metadataextraction_export flag is enabled

bugfix: renaming obsolete decision-export to decision-export-to-hbase

#1308 reverting uri:oozie:distcp-action:0.2 change: version is not properly recognized by oozie 3.3.2-cdh4.3.1

#1260 enabling document to protein databank reference extraction in primary workflow, supporting 3 new parameters: active_referenceextraction_pdb, export_action_set_id_document_pdb, export_referenceextraction_pdb_url_root

#1301 introducing explicit export mode flags: active_export_to_hbase and active_export_to_json. This way both exports can be enabled or both of them can be disabled.

#1308 switching distcp namespace to uri:oozie:distcp-action:0.2

disabling export by setting active_export flag to false. Results will be converted to JSON records

#1301 introducing common/export_to_json and utilizing this subworkflow in both primary and preprocessing workflows executing it when active_export=false which means hbase export is disabled

#118 explicitly defining input_document_websiteusage_similarity parameter. This is not a bug fix because exporter works properly without explicitly defining input port due to propagate-configuration mode but we should have all input port definitions aligned to avoid confusions.

bugfixing existing fault removal which was missing

bugfixing existing fault removal which was missing

setting false to remove_sideproducts, otherwise whole workingDir will be erased

#1257 dropping schema generation related hacks in all map-reduce modules, switching to literal schema parameters

updating job.properties, disabling all algorithms by default

#1248 fixing transition node name postprocessing-joining to merge-joining

#1248 introducing fault subdirectory support in all workflows wrapping metadataextraction subworkflow up to the processing and primary root workflows. This should prevent fault directory from being removed when ${remove_sideproducts} flag is enabled, it will be propagated along with metadata and plaintext.

updating job.properties

Removing usage of working_dir from Java workflow node.

#1195 removing obsolete ports docreation and datasetid from hbase mapred import, removing references to those ports in workflow.xml files, updating transformer by removing filtering by datasetid due to decisions made in #1072

#1176 defining remove_sideproducts property in workflows headers

#1176 introducing side products removal in common import by maintaining remove_sideproducts flag set to true by default.
Notice: do not provide any output directory location pointing to workingDir subdirectory!

removing duplicate collapser import and aligning worklfow definition

#1172 introducing support for active_export parameter in both preprocessing and primary workflows

updating job.properties

#1153 utilizing ${user.name} placeholder in ${workingDir} generation process, copying version.properties from oozie_app to mark execution environment with application version

#1147 introducing HTML import and HTML plaintext ingestion in main workflows: primary and preprocessing

updating job.properties

#1133 dropping useless workfing_dir creation for java nodes

#118 fixing typos

updating job.properties

#118 introducing website usage analysis as integral part of primary workflow

updating job.properties

#118 propagating configuration in main workflow.xml

#118 updating job.properties

comments added

#118 introducing mainworkflows_websiteusage_document_main workflow binding all subworkflows required to process logs and generate document similarities

#1083 enabling webcrawl ingester module extracting FX field from plaintext before executing project reference extraction

updating default job properties

#1070 updating import_project_concepts_context_ids_csv default value to "fet-fp7,fet-h2020"

#1070 introducing support for multiple context identifiers, replacing import_project_concepts_context_id IIS input parameter with import_project_concepts_context_ids_csv

updating job.properties

#1072 transition fix: replacing forking_skip_imported_data with export

#1072 dropping IIS feature filtering out already existing project and dataset references from IIS export

updating job.properties

updating job.properties

#919 renaming DocumentToResearchInitiative to DocumentToConceptId and DocumentToResearchInitiatives to DocumentToConceptIds

#1022 introducing PMC extracted document metadata collapser removing duplicates before sending output to PMC citation ingestion module

#919 enabling concepts matching for FET projects in mainworkflows: import, export, primary and preprocessing

#1017 accepting ExtractedDocumentMetadata instead of DocumentText at PMC citation ingestion input. Aliging integration test and importer workflow.

#1022 introducing extracted document metadata collapser at importing phase.
Propagating extracted document mentadata (including PMC ingested metadata) to processing part of workflow what can be exploited by citation matching module.
Introducing citations collapser in last stage of processing phase collapsing ingested citations with matched citations.

#1017 introducing new PMC metadata ingestion currently extracing references, journal and pages fields.
Replacing DOM/XPath based citations ingestion with much faster SAX version. Changing pmidtooaid transformer utilizing ExtractedDocumentMetadata instead of parsing XML file. Enabling PMC metadata ingestion in common/import.

#963 propagating dataset -> mdstore from import to exporting phase: importer produces DocumentToMDStore datasetore utilized by exporter module. Updating transformer definition to handle DocumentToMDStore instead of Identifier schema

updating job.properties

updating job.properties

updating job.properties

updating job.properties

updating job.properties

updating job.properties

updating job.properties

updating job.properties: adding metadataextraction_excluded_checksums=4f5cc34f137de4dc89766a9366ca66de,6495a568200b1cee40baa00072b1800a

updating job.properties

introducing support for active_existence_filter, set to true by default. Setting this parameter to false allows processing contents not having its counterpart among metadata records retrieved from HBase. This solution was required to e.g. process ubiquity contents which were not present in HBase dump metadata.

updating job.properties

renaming metadataextraction_excluded_ids to more appropriate metadataextraction_excluded_checksums

#913 introducing support for max file size parameter, currently checked against Content-Lenght header

updating job.properties

#757 hooking up ingest_pmc_idmapping_pmidtooaid subworkflow with mainworkflows/common/import. From now on citations are matched by pmid as well.

updating job.properties

#883 providing blacklisted_objectstores_csv input parameter set to $UNDEFINED$ value by default

updating job.properties

input port name fix: input_citation->input_citations

introducing merge_body_with_updates flag support in common/import, setting to true in statistics workflow

introducing regex support in result approver to support iis::* kind of provenance, updating workflow definitions with proper regex values

#840 moving IdentifierMapping from importer to common package

#840 renaming DeduplicationMapping to more generic IdentifierMapping

#757 aligning common importer with current API of PMC citations ingestion

#637 renaming document_extractedMetadata algorithm to more descriptive document_affiliations, propagating changes to action set identifier properties names

reenabling PMC ingestion when citationmatching flag is set

updating job properties

setting export_action_set_id_entity_dataset to $UNDEFINED$ by default, this should not be required because dataset reference extraction module might be deactivated. Check will be performed at dataset entity exporter module and when value is not set - exception will be raised.

#757 temporarily disabling PMC ingestion until fixing openaire identifiers building process

#568, #577 enabling proper citations export by introducing PMC citation ingestion and citation matching outcome merging and grouping for exporting purposes. Introducing union instead of collapser which should be introduced in near future.

removing output_citation_pmc port duplicate

removing common import input parameters which are not required in this context

introducing trust_level_threshold support in statistics workflow

introducing trust_level_threshold support in common import workflow

providing default value for action_set_id_entity_dataset set to $UNDEFINED$. This change is required when exporting in statistics export mode where no entities are exported and such parameter should not be required.

introducing dedicated statistics mainworkflow encapsulating importing, processing and exporting phases. This workflow was introduced explicitly for statistics purposes because we want to operate over InformationSpace imported data in contrary to primary workflow where some of the statistics input was inferred and it wasn't clear whether it will become part of InformationSpace.