D-Net

Need support? Contact us via email at: dnet-team@isti.cnr.it

This webapp contains the minimal set of services needed to feature:

- Collection of metadata records in oai_dc format via OAI-PMH, FTP, local file system, HTTP.

- Indexing of DMF records in a Solr full-text index

- OAI-PMH export of aggregated metadata records in DMF and oai_dc formats. More formats can be added at runtime by providing a dedicated XSLT from DMF to the desired target format.

- Operating system: almost anything but Windows

- HARD DISK space: mostly depends on the quantity and size of records you are going to collect. A couple of GBs for a small repository (<10K metadata recods) should be fine. See suggestions on installing mongodb below.

## Software requirements

* Mongodb >= 2.4: used to store the collected and transformed metadata records. Each collected record will be stored in three separate "versions": original, transformed, pmh-ready, hence enough disk space should be available for mongoDB.

* Solr 4.9.x or 4.10.x: used to make the documents searchable. The solr server should be run using the option '-DzkRun' to instruct solr to start the zookeeper server.

Note that Tomcat, Solr and Mongodb can be installed in the same machine or in dedicated nodes, although this requires to change some default system properties.

#Running the D-Net web app with Maven

Either if you want to run the D-Net web app with the Tomcat7 plugin for maven, or you want to build the .war file to deploy on a running tomcat,

The D-Net Software is developed in Java using Maven. You can try out the D-Net web app on your local machine with the tomcat7 plugin, provided you are also running a mongodb and a solr server on localhost that are listening to the relative standard ports.

Please note that the solr client used in D-Net needs to interact with the zookeeper server. For simplicity we suggest to use the embedded zookepper instance provided within the solr distribution. By default solr listens on the 8983 port and its embedded zookeeper server on the 9983 port.

To override properties, you can modify <code>dnet-basic-aggregator/src/main/resources/eu/dnetlib/cnr-site.properties</code>. Please check the Section D-Net Configuration and the PROPERTIES.md file for more information about D-Net properties.

```

> cd dnet-basic-aggregator

The webapp should be ready and running at

```

If you want to build the web app yourself, then keep reading...

 > cd dnet-basic-aggregator

The <code>.war</code> file is then created into the <code>target</code> directory.

#D-Net configuration

</br>Example: <code>container.hostname = dnet-host.dnet.eu</code>

- <code>container.port</code>: the port where the web app will be running. Default is 8280.

</br>Example: <code>container.port = 8080</code>

- <code>container.context</code>: the name of the web app (i.e. the name of the war file). Default is "app". The default value should *only* be used in local development scenarios.

</br>Example: <code>container.context = is</code>

- <code>dnet.data.path</code>: path to the directory where all D-Net related resources will be saved. An embedded existDB will be automatically installed in this directory during the first start-up. The directory must be writable by the user running tomcat. Default value is <code>/tmp/dnet</code>. The default value should *only* be used in local development scenarios.

</br>Example: <code>dnet.data.path = /var/lib/dnet</code>

- <code>services.aggregator.country</code>: your country code. Default is <code>EU</code> (Europe).

</br>Example: <code>services.aggregator.country = IT</code>

- <code>services.aggregator.name</code>: the name of your aggregator. Default is "D-NET"

</br>Example: <code>services.aggregator.name = TEST_Aggregator</code>.

- <code>services.mdstore.mongodb.host</code>: the machine hosting mongodb for the storage of metadata records (M[eta]D[ata]Store). Default is <code>localhost</code>.

</br>Example: <code>services.mdstore.mongodb.db = mdstore_1</code>

- <code>dnet.logger.mongo.host</code>: the machine hosting mongodb for the storage of workflow logs. Default is localhost.

</br>Example: <code>dnet.logger.mongo.host = mongo.dnet.eu</code>

- <code>dnet.logger.mongo.db</code>: name of the mongodb database to be used for the storage of workflow logs. Default is "dnet_logs_minimal".

</br>Example: <code>dnet.logger.mongo.db = dnet_logs_1</code>

- <code>services.oai.publisher.repo.name</code>: name of the OAI-PMH Publisher, as it will appear in the OAI Identify response. Default is "D-Net OAI-PMH Publisher".

</br>Example: <code>services.oai.publisher.repo.name = TEST_Aggregator OAI-PMH Publisher</code>

- <code>services.oai.publisher.repo.email</code>: email of the OAI-PMH Publisher administrator, as it will appear in the OAI Identify response. Default is "dnet-admin@mock.it". The default *must not* be used in beta or production system for it is a mock email.

</br>Example: <code>name.surname@valid.mail.com</code>

- <code>dnet.admin.password</code>: md5sum of the password that will allow the user "admin" to login to the D-Net Admin UI. To generate the new password: <code>echo -n "thePassword" | md5sum</code>. Default is "dnet-minimal" (without double quotes). The default value *should always be overridden*.

- <code>service.solr.index.jsonConfiguration</code>: information about the Solr instance to be used to create full-text indices on the aggregated metadata records. Default value assumes a local Solr instance. Specifically:

{"id":"solr", "address":"localhost:9983", "port":"8983", "webContext":"solr", "numShards":"1", "replicationFactor":"1", "host":"localhost",	"feedingShutdownTolerance":"30000",	"feedingBufferFlushThreshold":"1000", "feedingSimulationMode":"false", "luceneMatchVersion":"4.9",	"serverLibPath":"../../../../contrib/extraction/lib", "filterCacheSize":"512","filterCacheInitialSize":"512",	"queryCacheSize":"512","queryCacheInitialSize":"512", "documentCacheSize":"512", "documentCacheInitialSize":"512", "ramBufferSizeMB":"960","mergeFactor":"40",	"autosoftcommit":"-1","autocommit":"15000", "termIndexInterval":"1024","maxIndexingThreads":"8", "queryResultWindowSize":"20","queryResultMaxDocCached":"200"}

</code>

#Using D-Net

Under the root folder of the project you can find the folder <code>mock-repository-content</code>.

* Access the Admin UI (http://${container.hostname}:${container.port}/${container.context}/mvc/ui/index.do)

* Go on Datasource Management --> Overview and search for "mock"

* Click on the meta-workflow "Collection and Transformation" and configure its workflows with the missing parameter for the transformation rule

* Ensure the launch mode is set to "Auto" for each workflow

* Wait for all the workflows to complete: collect, transform, index, oai, and oaiPostFeed

* Verify that the records get transformed and indexed: click on MD Inspectors --> D-Net content checker and perform some queries

	* http://${container.hostname}:${container.port}/${container.context}/mvc/oai/oai.do?verb=ListRecords&metadataPrefix=dmf for the DMF metadata format