1
|
# D-Net Software Toolikt
|
2
|
|
3
|
This is a minimal instance of the D-Net software toolkit, a software framework for the realization of aggregative data infrastructures.
|
4
|
|
5
|
Official Web Site: http://www.d-net.research-infrastructures.eu/
|
6
|
|
7
|
Features:
|
8
|
|
9
|
- Collection of metadata records in oai_dc format via OAI-PMH, xxx
|
10
|
|
11
|
- Transformation of the collected metadata records into an internal format named DMF (Driver Metadata Format)
|
12
|
|
13
|
- Indexing of DMF records into a Solr full-text index
|
14
|
|
15
|
- OAI-PMH exports of aggregated metadata records in DMF and oai_dc formats. More formats can be added at runtime by providing a dedicated XSLT from DMF to the desired target format.
|
16
|
|
17
|
Any questions, suggestions and feedback can be sent to dnet-team@isti.cnr.it.
|
18
|
|
19
|
# Installation requirements
|
20
|
This minimal instance can be run on a single machine as web application to be deployed on a Tomcat container.
|
21
|
## Hardware requirements
|
22
|
|
23
|
Suggested minimal hardware requirements:
|
24
|
- SO
|
25
|
- CPU
|
26
|
- RAM
|
27
|
- HARD DISK space: heavily depends on the quantity and size of records you are going to harvest. See suggestions on instaling mongodb below.
|
28
|
|
29
|
## Software requirements
|
30
|
Software required:
|
31
|
|
32
|
* Apache Tomcat 7: the webapp container
|
33
|
* Mongodb >= 2.4: used to store the collected and transformed metadata records. Consider that each collected record will be stored in three separate "versions": original, transformed, pmh-ready, hence enough disk space should be available for mongoDB.
|
34
|
* Solr >= 4.9
|
35
|
|
36
|
Note that Tomcat, Solr and Mongodb can be installed in the same machine or in three separate machines.
|
37
|
|
38
|
#D-Net configuration
|
39
|
Before you start the web application, you need to configure at least the following properties.
|
40
|
For the full list of available properties and their values, check PROPERTIES.md.
|
41
|
|
42
|
Create a file named <code>cnr.override.properties</code> in <code>$yourTomcatHomeDirectory$/common/classes</code> (<code>$yourTomcatHomeDirectory$</code> will likely be something similar to <code>/var/lib/tomcat7</code>)
|
43
|
|
44
|
- <code>container.hostname</code>: the host name where the web app will be running. Default value is <code>localhost</code>. The default value should *only* be used in local development scenarios.
|
45
|
Example: <code>container.hostname = dnet-host.dnet.eu</code>
|
46
|
- <code>container.port</code>: the port where the web app will be running. Default is 8280.
|
47
|
Example: <code>container.port = 8080</code>
|
48
|
- <code>container.context</code>: the name of the web app (i.e. the name of the war file). Default is "app". The default value should *only* be used in local development scenarios.
|
49
|
Example: <code>container.context = is</code>
|
50
|
- <code>dnet.data.path</code>: path to the directory where all D-Net related resources will be saved. An embedded existDB will be automatically installed in this directory during the first start-up. The directory must be writable by the user running tomcat. Default value is <code>/tmp/dnet</code>. The default value should *only* be used in local development scenarios.
|
51
|
Example: <code>dnet.data.path = /var/lib/dnet</code>
|
52
|
- <code>services.aggregator.country</code>: your country code. Default is <code>EU</code> (Europe).
|
53
|
Example: <code>services.aggregator.country = IT</code>
|
54
|
- <code>services.aggregator.name</code>: the name of your aggregator. Default is "D-NET"
|
55
|
Example: <code>services.aggregator.name = TEST_Aggregator</code>.
|
56
|
- <code>services.mdstore.mongodb.host</code>: the machine hosting mongodb for the storage of metadata records (M[eta]D[ata]Store). Default is localhost.
|
57
|
Example: <code>services.mdstore.mongodb.host = mongo.dnet.eu</code>
|
58
|
- <code>services.mdstore.mongodb.db</code>: name of the mongodb database to be used for the storage of metadata records. Default is "mdstore_minimal".
|
59
|
Example: <code>services.mdstore.mongodb.db = mdstore_1</code>
|
60
|
- <code>dnet.logger.mongo.host</code>: the machine hosting mongodb for the storage of workflow logs. Default is localhost.
|
61
|
Example: <code>dnet.logger.mongo.host = mongo.dnet.eu</code>
|
62
|
- <code>dnet.logger.mongo.db</code>: name of the mongodb database to be used for the storage of workflow logs. Default is "dnet_logs_minimal".
|
63
|
Example: <code>dnet.logger.mongo.db = dnet_logs_1</code>
|
64
|
- <code>services.oai.publisher.repo.name</code>: name of the OAI-PMH Publisher, as it will appear in the OAI Identify response. Default is "D-Net OAI-PMH Publisher".
|
65
|
Example: <code>services.oai.publisher.repo.name = TEST_Aggregator OAI-PMH Publisher</code>
|
66
|
- <code>services.oai.publisher.repo.email</code>: email of the OAI-PMH Publisher administrator, as it will appear in the OAI Identify response. Default is "dnet-admin@mock.it". The default *must not* be used in beta or production system for it is a mock email.
|
67
|
Example: <code>name.surname@valid.mail.com</code>
|
68
|
- <code>dnet.admin.password</code>: md5sum of the password that will allow the user "admin" to login to the D-Net Admin UI. To generate the new password: <code>echo thePassword -n | md5</code>. Default is "dnet-minimal" (without double quotes). The default value *should always be overridden*.
|
69
|
Example: <code>dnet.admin.password = 5d1ed3888708c0f4cd46b29306a6b449</code>, where 5d1ed3888708c0f4cd46b29306a6b449 is the md5 for the string "pwd" obtained via the command <code>echo pwd -n | md5</code>.
|
70
|
- <code>service.solr.index.jsonConfiguration</code>: information about the Solr instance to be used to create full-text indices on the aggregated metadata records. Default value assumes a local Solr instance. Specifically:
|
71
|
<code>
|
72
|
{"id":"solr",\
|
73
|
"address":"localhost:9983",\
|
74
|
"port":"8983",\
|
75
|
"webContext":"solr",\
|
76
|
"numShards":"1",\
|
77
|
"replicationFactor":"1",\
|
78
|
"host":"localhost",\
|
79
|
"feedingShutdownTolerance":"30000",\
|
80
|
"feedingBufferFlushThreshold":"1000",\
|
81
|
"feedingSimulationMode":"false",\
|
82
|
"luceneMatchVersion":"4.9",\
|
83
|
"serverLibPath":"../../../../contrib/extraction/lib",\
|
84
|
"filterCacheSize":"512","filterCacheInitialSize":"512",\
|
85
|
"queryCacheSize":"512","queryCacheInitialSize":"512",\
|
86
|
"documentCacheSize":"512","documentCacheInitialSize":"512",\
|
87
|
"ramBufferSizeMB":"960","mergeFactor":"40",\
|
88
|
"autosoftcommit":"-1","autocommit":"15000",\
|
89
|
"termIndexInterval":"1024","maxIndexingThreads":"8",\
|
90
|
"queryResultWindowSize":"20","queryResultMaxDocCached":"200"}
|
91
|
</code>
|
92
|
|
93
|
If you are not running the Solr service on the same machine where Tomcat runs, then you need to override the above configuration according to your Solr server installation.
|
94
|
Typically, changing <code>address</code> and <code>host</code> is enough if your Solr server is not configured for sharding and replication.
|
95
|
For more details refer to the Solr documentation.
|
96
|
|
97
|
|
98
|
# Launching the D-Net web app
|
99
|
The D-Net Software is developed in Java with Maven.
|
100
|
In this distribution you will find a ready-to-deploy war package.
|
101
|
|
102
|
Copy the war file into the Tomcat 7 <code>webapps</code> directory, ensure you have overridden the properties as explained in the previous section and restart Tomcat.
|
103
|
|
104
|
The webapp should be running at http://${container.hostname}:${container.port}/${container.context}
|
105
|
|
106
|
If you want to build the web app yourself, then keep reading...
|
107
|
|
108
|
## Maven settings
|
109
|
|
110
|
If you want to run the D-Net web app with the Tomcat7 plugin for maven or if you want to build the .war file to deploy on a running tomcat,
|
111
|
then you should have installed maven3 and you must add the following repository into your <code>settings.xml</code>:
|
112
|
|
113
|
```
|
114
|
<repository>
|
115
|
<id>dnet-bootstrap-releases</id>
|
116
|
<name>D-Net Bootstrap Releases</name>
|
117
|
<url>http://maven.research-infrastructures.eu/nexus/content/repositories/dnet4-bootstrap-release/</url>
|
118
|
<releases>
|
119
|
<enabled>true</enabled>
|
120
|
</releases>
|
121
|
<snapshots>
|
122
|
<enabled>false</enabled>
|
123
|
</snapshots>
|
124
|
<layout>default</layout>
|
125
|
</repository>
|
126
|
```
|
127
|
|
128
|
We also suggest to add the Tomcat plugin to the plugins group at the bottom of the same file:
|
129
|
|
130
|
```
|
131
|
<pluginGroups>
|
132
|
<pluginGroup>org.apache.tomcat.maven</pluginGroup>
|
133
|
</pluginGroups>
|
134
|
```
|
135
|
|
136
|
|
137
|
## Testing on local machine:
|
138
|
The D-Net Software is developed in Java with Maven. You can try out the D-Net web app on your local machine with the tomcat7 plugin.
|
139
|
```
|
140
|
> cd dnet-minimal-container
|
141
|
|
142
|
> mvn tomcat7:run
|
143
|
```
|
144
|
|
145
|
The webapp should be running at http://localhost:8280/app .
|
146
|
|
147
|
## Deployment on a Tomcat instance
|
148
|
The D-Net Software is developed in Java with Maven.
|
149
|
|
150
|
To build the war to use in a Tomcat 7 web app container:
|
151
|
|
152
|
```
|
153
|
> cd dnet-minimal-container
|
154
|
|
155
|
> mvn package
|
156
|
```
|
157
|
|
158
|
The .war file is then created into the <code>target</code> directory.
|
159
|
|
160
|
|
161
|
|