D-Net

updating job.properties

#1153 utilizing ${user.name} placeholder in ${workingDir} generation process, copying version.properties from oozie_app to mark execution environment with application version

#1147 introducing HTML import and HTML plaintext ingestion in main workflows: primary and preprocessing

#1147 renaming icm-iis-ingest-webcrawl module to icm-iis-ingest to make it more generic so it could contain not only webcrawl related ingesters but html ingesters as well

updating job.properties

overriding memory parameter due to test cluster memory limitations, setting it to Xmx256m

overriding memory parameter due to test cluster memory limitations, setting it to Xmx128m

overriding memory parameter due to test cluster memory limitations, setting it to Xmx512m

updating expected classes in integration test after recent #720 change and fixing confidence level distribution

overriding memory parameter due to test cluster memory limitations

#1133 dropping useless workfing_dir creation for java nodes

#1038 introducing ranges in dependencies definition for all IIS modules

#118 fixing typos

updating job.properties

#118 introducing website usage analysis as integral part of primary workflow

updating job.properties

#118 propagating configuration in main workflow.xml

introducing explicitly defined icm-iis-schemas SNAPSHOT dependency to prevent resolving earlier, released transitive version

#118 upgrading IIS dependencies to most recent snapshots

#118 updating job.properties

#118 introducing uoa-iis-websiteusage dependency in mainworkflows

comments added

#118 introducing mainworkflows_websiteusage_document_main workflow binding all subworkflows required to process logs and generate document similarities

#1083 enabling webcrawl ingester module extracting FX field from plaintext before executing project reference extraction

updating default job properties

#720 fixing document classification algorithm confidence level distribution, switching mainworkflows pom dependency to the fixed document classification snapshot

#1070 updating import_project_concepts_context_ids_csv default value to "fet-fp7,fet-h2020"

#1070 introducing support for multiple context identifiers, replacing import_project_concepts_context_id IIS input parameter with import_project_concepts_context_ids_csv

updating job.properties

#1072 transition fix: replacing forking_skip_imported_data with export

#1072 dropping IIS feature filtering out already existing project and dataset references from IIS export

#1065 upgrading icm-iis-parent-container dependency from 1.0.0 to 1.0.1-SNAPSHOT after introducing FCT support

#1065 upgrading uoa-iis-referenceextraction dependency from 1.0.0 to 1.0.1-SNAPSHOT after introducing FCT support

[maven-release-plugin] prepare for next development iteration

[maven-release-plugin] prepare release icm-iis-mainworkflows-1.0.0

changing snapshot dependencies to released ones

#1044 upgrading dependencies to released versions and parent version to most recent snapshot for unreleased modules

introducing scm definition

updating job.properties

updating job.properties

#919 renaming DocumentToResearchInitiative to DocumentToConceptId and DocumentToResearchInitiatives to DocumentToConceptIds

#1022 introducing PMC extracted document metadata collapser removing duplicates before sending output to PMC citation ingestion module

#919 adding missing i/o ports related to FET projects reference extraction

#919 enabling concepts matching for FET projects in mainworkflows: import, export, primary and preprocessing

#1017 accepting ExtractedDocumentMetadata instead of DocumentText at PMC citation ingestion input. Aliging integration test and importer workflow.

#1022 introducing extracted document metadata collapser at importing phase.
Propagating extracted document mentadata (including PMC ingested metadata) to processing part of workflow what can be exploited by citation matching module.
Introducing citations collapser in last stage of processing phase collapsing ingested citations with matched citations.

#1017 introducing new PMC metadata ingestion currently extracing references, journal and pages fields.
Replacing DOM/XPath based citations ingestion with much faster SAX version. Changing pmidtooaid transformer utilizing ExtractedDocumentMetadata instead of parsing XML file. Enabling PMC metadata ingestion in common/import.

#963 propagating dataset -> mdstore from import to exporting phase: importer produces DocumentToMDStore datasetore utilized by exporter module. Updating transformer definition to handle DocumentToMDStore instead of Identifier schema

introducing separate citations json containing expected results, not enabled in workflow yet

updating job.properties

updating job.properties

updating job.properties

updating job.properties

updating job.properties

updating job.properties

updating job.properties

updating job.properties: adding metadataextraction_excluded_checksums=4f5cc34f137de4dc89766a9366ca66de,6495a568200b1cee40baa00072b1800a

updating job.properties

introducing support for active_existence_filter, set to true by default. Setting this parameter to false allows processing contents not having its counterpart among metadata records retrieved from HBase. This solution was required to e.g. process ubiquity contents which were not present in HBase dump metadata.

fixing citations schema type

updating job.properties

renaming metadataextraction_excluded_ids to more appropriate metadataextraction_excluded_checksums

#913 introducing support for max file size parameter, currently checked against Content-Lenght header

adding integration-test job name suffix

adding icm-iis-mainworkflows_import entry

setting nigtly parameter

updating job.properties

enabling document classification and reserach initiatives reference extraction algorithms

#757 hooking up ingest_pmc_idmapping_pmidtooaid subworkflow with mainworkflows/common/import. From now on citations are matched by pmid as well.

updating profiles names

fixing job name for integration test

updating job.properties

#883 providing blacklisted_objectstores_csv input parameter set to $UNDEFINED$ value by default

updating job.properties

input port name fix: input_citation->input_citations

introducing merge_body_with_updates flag support in common/import, setting to true in statistics workflow

introducing regex support in result approver to support iis::* kind of provenance, updating workflow definitions with proper regex values

#840 moving IdentifierMapping from importer to common package

#840 renaming DeduplicationMapping to more generic IdentifierMapping

#757 aligning common importer with current API of PMC citations ingestion

disabling workflow tests

introducing external-integration-test: iis/mainworkflows/integration/primary/processing entry

#637 renaming document_extractedMetadata algorithm to more descriptive document_affiliations, propagating changes to action set identifier properties names

introducing cloudera repository in parent container, removing repository definitions from individual IIS modules

removing extracted_metadata.json which will not be checked anymore

reenabling PMC ingestion when citationmatching flag is set

updating job properties

skipping extracted_metadata comparison which is cumbersome due to frequent changes and large volume of references

introducing newly added address field in json record

fixing field names after recent Affiliation.avdl refactoring and adding countryCode field, renaming contry to countryName

setting export_action_set_id_entity_dataset to $UNDEFINED$ by default, this should not be required because dataset reference extraction module might be deactivated. Check will be performed at dataset entity exporter module and when value is not set - exception will be raised.

#757 temporarily disabling PMC ingestion until fixing openaire identifiers building process

#568, #577 enabling proper citations export by introducing PMC citation ingestion and citation matching outcome merging and grouping for exporting purposes. Introducing union instead of collapser which should be introduced in near future.

updating expected output

removing output_citation_pmc port duplicate

updating performance test

moving ACM importer to icm-iis-mainworkflows due to extending dependances with cermine, introducing performance tests

removing common import input parameters which are not required in this context