D-Net

[maven-release-plugin] copy for tag dnet-modular-collector-service-3.3.9

attempt to fix the CSV files with invalid quotes

tests

BOM management

fileCSV plugin now support quotes as requested in #2108

CSV format can trim

added more badEntities, to be fixed on collection phase (#2178)

setSpec and setName visualization in ListSets forms

Ingore test that uses remote SFTP

removed reference to the not anymore used DateTimeFormatter for jsch dates

Better using integer mtime. Parsing string dates is hell.

Ignore tests that use remote sftp locations

SFTP plugin now supports incremental collection based on the last modified date of files.

changed basePath parameter optional in plugin filesfrommetadata

HttpConnector complemented with CookieHandler, CookieManager due to issue with URV data provider

user-agent string changed again due to issue with yet another data provider

add the same parameter to protocol files_from_mdstore

Ignore test using remote ftp

fixed a typo in two comments (FTP vs SFTP)

- removed old ftp (code and spring bean)
- renamed new ftp (aka ftp2) as ftp
- added a first version of sftp protocol for collection
- minor changes in the code (e.g. exception classes involved)

ups, wrongly removal of xmlns

basePath access param set as mandatory for files_from_metadata. Available path to choose from should be in service parameters of the Object store service (property key: 'basePath')

added User-Agent request property to circumvent http 403 status code, example kagoshima repository

clean a log

fix some NPE and IndexOutOfBoundException in CSV plugins

solving ticket #1160

Created a collectorServiceErrorMessageLogList to help jochen in debugging OAI harvests gone wrong.

returning error code & message when a timeout occurs

set log to debug

trying to fix ftp collector service - test fixed

trying to fix ftp collector service

improved logs

Avoid noisy info logs and throw UnsupportedOperation when needed.

reindent

Test for HttpConnector

implemented layer file system for Object Store Service

Implements 1236#note-16: stop reading pages if I have already seen ALL sets provided by the current page.

Testing

Preventing to cycle over ListSets pages when the publishers do not work and return always the same page.

added extensions filter to ftp2 plugin
-test corrected.

added extensions filter to ftp2 plugin

ftp2 failures in disconnection/logout do not invalidate the WF (an error is logged, though)

301/302 codes are not manually handled

test

fixed a bean name

fixed test

httpList plugin

added mongodump collector plugin

corrected a bug in next()

added contructors with File argument

added tests for zip and tar.gz iterators

tests re-packaged

package refactoring

first implementation of the zip collector plugin

removed RuntimeExceptions
better logging

logger removed

removed extension param

First implementation of .tar.gz plugin

Filesystem plugin' tests updated

removed useless toString()

laxy and recursive FS collector plugin.
tested on datacite.

-Extensions param is a CSV list e.g "xml,pg,pdf"
-The behavior is by design recursive.

fixed: wrong isBlank condition testing for filter presence

fixed a test

oai parameter order

replaced required with optional

reimplemented funtion to obtain valid values for a param

reimplemented the ProtocolDescriptor

Added bean for GZip collector

Implemented collector plugin for gz files.

Launching exception when needed

Properties for httpConnector

Increase READ_TIMEOUT

encoding of oai parameters

ignoring test that depends on an external FTP site

plugin to harvest a list o oai sets

noRecordsMatch management

Added new collector plugin

Added new colletor plugin for CSV retreived from HTTP URL

removed unused import

alternative ftp plugin (use "ftp2")

implemented a new version of FileSystem collector plugin which uses a file system library of java 1.7

fixed package

deleted invalid package

moved a test in the right package

-introduced readTimeout for oai harvester.
default is set to 60s

-the returned XML document is inspected for errors.
e.g. bad resumption token is not considered

added closeQuitely()

stream closed

retry on exceptions

javadoc annotations

Refactoring

oai harvester using Jochen's HttpConnector and XmlCleaner

removed the exception when a param is missing

fixed merging with branches csv

new version of ListProtocols

merged branch NewCSVPlugin

sysout transformed into log