1 |
36918
|
alessia.ba
|
# D-Net Software Toolikt
|
2 |
|
|
|
3 |
|
|
This is a minimal instance of the D-Net software toolkit, a software framework for the realization of aggregative data infrastructures.
|
4 |
|
|
|
5 |
|
|
Official Web Site: http://www.d-net.research-infrastructures.eu/
|
6 |
|
|
|
7 |
|
|
Features:
|
8 |
|
|
|
9 |
|
|
- Collection of metadata records in oai_dc format via OAI-PMH, xxx
|
10 |
|
|
|
11 |
|
|
- Transformation of the collected metadata records into an internal format named DMF (Driver Metadata Format)
|
12 |
|
|
|
13 |
|
|
- Indexing of DMF records into a Solr full-text index
|
14 |
|
|
|
15 |
|
|
- OAI-PMH exports of aggregated metadata records in DMF and oai_dc formats. More formats can be added at runtime by providing a dedicated XSLT from DMF to the desired target format.
|
16 |
|
|
|
17 |
|
|
Any questions, suggestions and feedback can be sent to dnet-team@isti.cnr.it.
|
18 |
|
|
|
19 |
|
|
# Installation requirements
|
20 |
|
|
This minimal instance can be run on a single machine as web application to be deployed on a Tomcat container.
|
21 |
|
|
## Hardware requirements
|
22 |
|
|
|
23 |
|
|
Suggested minimal hardware requirements:
|
24 |
|
|
- SO
|
25 |
|
|
- CPU
|
26 |
|
|
- RAM
|
27 |
|
|
- HARD DISK space: heavily depends on the quantity and size of records you are going to harvest. See suggestions on instaling mongodb below.
|
28 |
|
|
|
29 |
|
|
## Software requirements
|
30 |
|
|
Software required:
|
31 |
|
|
|
32 |
|
|
* Apache Tomcat 7: the webapp container
|
33 |
|
|
* Mongodb >= 2.4: used to store the collected and transformed metadata records. Consider that each collected record will be stored in three separate "versions": original, transformed, pmh-ready, hence enough disk space should be available for mongoDB.
|
34 |
|
|
* Solr >= 4.9
|
35 |
|
|
|
36 |
|
|
Note that Tomcat, Solr and Mongodb can be installed in the same machine or in three separate machines.
|
37 |
|
|
|
38 |
|
|
#D-Net configuration
|
39 |
|
|
Before you start the web application, you need to configure at least the following properties.
|
40 |
|
|
For the full list of available properties and their values, check PROPERTIES.md.
|
41 |
|
|
|
42 |
|
|
Create a file named <code>cnr.override.properties</code> in <code>$yourTomcatHomeDirectory$/common/classes</code> (<code>$yourTomcatHomeDirectory$</code> will likely be something similar to <code>/var/lib/tomcat7</code>)
|
43 |
|
|
|
44 |
|
|
- <code>container.hostname</code>: the host name where the web app will be running. Default value is <code>localhost</code>. The default value should *only* be used in local development scenarios.
|
45 |
|
|
Example: <code>container.hostname = dnet-host.dnet.eu</code>
|
46 |
|
|
- <code>container.port</code>: the port where the web app will be running. Default is 8280.
|
47 |
|
|
Example: <code>container.port = 8080</code>
|
48 |
|
|
- <code>container.context</code>: the name of the web app (i.e. the name of the war file). Default is "app". The default value should *only* be used in local development scenarios.
|
49 |
|
|
Example: <code>container.context = is</code>
|
50 |
|
|
- <code>dnet.data.path</code>: path to the directory where all D-Net related resources will be saved. An embedded existDB will be automatically installed in this directory during the first start-up. The directory must be writable by the user running tomcat. Default value is <code>/tmp/dnet</code>. The default value should *only* be used in local development scenarios.
|
51 |
|
|
Example: <code>dnet.data.path = /var/lib/dnet</code>
|
52 |
|
|
- <code>services.aggregator.country</code>: your country code. Default is <code>EU</code> (Europe).
|
53 |
|
|
Example: <code>services.aggregator.country = IT</code>
|
54 |
|
|
- <code>services.aggregator.name</code>: the name of your aggregator. Default is "D-NET"
|
55 |
|
|
Example: <code>services.aggregator.name = TEST_Aggregator</code>.
|
56 |
|
|
- <code>services.mdstore.mongodb.host</code>: the machine hosting mongodb for the storage of metadata records (M[eta]D[ata]Store). Default is localhost.
|
57 |
|
|
Example: <code>services.mdstore.mongodb.host = mongo.dnet.eu</code>
|
58 |
|
|
- <code>services.mdstore.mongodb.db</code>: name of the mongodb database to be used for the storage of metadata records. Default is "mdstore_minimal".
|
59 |
|
|
Example: <code>services.mdstore.mongodb.db = mdstore_1</code>
|
60 |
|
|
- <code>dnet.logger.mongo.host</code>: the machine hosting mongodb for the storage of workflow logs. Default is localhost.
|
61 |
|
|
Example: <code>dnet.logger.mongo.host = mongo.dnet.eu</code>
|
62 |
|
|
- <code>dnet.logger.mongo.db</code>: name of the mongodb database to be used for the storage of workflow logs. Default is "dnet_logs_minimal".
|
63 |
|
|
Example: <code>dnet.logger.mongo.db = dnet_logs_1</code>
|
64 |
|
|
- <code>services.oai.publisher.repo.name</code>: name of the OAI-PMH Publisher, as it will appear in the OAI Identify response. Default is "D-Net OAI-PMH Publisher".
|
65 |
|
|
Example: <code>services.oai.publisher.repo.name = TEST_Aggregator OAI-PMH Publisher</code>
|
66 |
|
|
- <code>services.oai.publisher.repo.email</code>: email of the OAI-PMH Publisher administrator, as it will appear in the OAI Identify response. Default is "dnet-admin@mock.it". The default *must not* be used in beta or production system for it is a mock email.
|
67 |
|
|
Example: <code>name.surname@valid.mail.com</code>
|
68 |
|
|
- <code>dnet.admin.password</code>: md5sum of the password that will allow the user "admin" to login to the D-Net Admin UI. To generate the new password: <code>echo thePassword -n | md5</code>. Default is "dnet-minimal" (without double quotes). The default value *should always be overridden*.
|
69 |
|
|
Example: <code>dnet.admin.password = 5d1ed3888708c0f4cd46b29306a6b449</code>, where 5d1ed3888708c0f4cd46b29306a6b449 is the md5 for the string "pwd" obtained via the command <code>echo pwd -n | md5</code>.
|
70 |
|
|
- <code>service.solr.index.jsonConfiguration</code>: information about the Solr instance to be used to create full-text indices on the aggregated metadata records. Default value assumes a local Solr instance. Specifically:
|
71 |
|
|
<code>
|
72 |
|
|
{"id":"solr",\
|
73 |
|
|
"address":"localhost:9983",\
|
74 |
|
|
"port":"8983",\
|
75 |
|
|
"webContext":"solr",\
|
76 |
|
|
"numShards":"1",\
|
77 |
|
|
"replicationFactor":"1",\
|
78 |
|
|
"host":"localhost",\
|
79 |
|
|
"feedingShutdownTolerance":"30000",\
|
80 |
|
|
"feedingBufferFlushThreshold":"1000",\
|
81 |
|
|
"feedingSimulationMode":"false",\
|
82 |
|
|
"luceneMatchVersion":"4.9",\
|
83 |
|
|
"serverLibPath":"../../../../contrib/extraction/lib",\
|
84 |
|
|
"filterCacheSize":"512","filterCacheInitialSize":"512",\
|
85 |
|
|
"queryCacheSize":"512","queryCacheInitialSize":"512",\
|
86 |
|
|
"documentCacheSize":"512","documentCacheInitialSize":"512",\
|
87 |
|
|
"ramBufferSizeMB":"960","mergeFactor":"40",\
|
88 |
|
|
"autosoftcommit":"-1","autocommit":"15000",\
|
89 |
|
|
"termIndexInterval":"1024","maxIndexingThreads":"8",\
|
90 |
|
|
"queryResultWindowSize":"20","queryResultMaxDocCached":"200"}
|
91 |
|
|
</code>
|
92 |
|
|
|
93 |
|
|
If you are not running the Solr service on the same machine where Tomcat runs, then you need to override the above configuration according to your Solr server installation.
|
94 |
|
|
Typically, changing <code>address</code> and <code>host</code> is enough if your Solr server is not configured for sharding and replication.
|
95 |
|
|
For more details refer to the Solr documentation.
|
96 |
|
|
|
97 |
|
|
|
98 |
|
|
# Launching the D-Net web app
|
99 |
|
|
The D-Net Software is developed in Java with Maven.
|
100 |
|
|
In this distribution you will find a ready-to-deploy war package.
|
101 |
|
|
|
102 |
|
|
Copy the war file into the Tomcat 7 <code>webapps</code> directory, ensure you have overridden the properties as explained in the previous section and restart Tomcat.
|
103 |
|
|
|
104 |
|
|
The webapp should be running at http://${container.hostname}:${container.port}/${container.context}
|
105 |
|
|
|
106 |
|
|
If you want to build the web app yourself, then keep reading...
|
107 |
|
|
|
108 |
|
|
## Maven settings
|
109 |
|
|
|
110 |
|
|
If you want to run the D-Net web app with the Tomcat7 plugin for maven or if you want to build the .war file to deploy on a running tomcat,
|
111 |
|
|
then you should have installed maven3 and you must add the following repository into your <code>settings.xml</code>:
|
112 |
|
|
|
113 |
|
|
```
|
114 |
|
|
<repository>
|
115 |
|
|
<id>dnet-bootstrap-releases</id>
|
116 |
|
|
<name>D-Net Bootstrap Releases</name>
|
117 |
|
|
<url>http://maven.research-infrastructures.eu/nexus/content/repositories/dnet4-bootstrap-release/</url>
|
118 |
|
|
<releases>
|
119 |
|
|
<enabled>true</enabled>
|
120 |
|
|
</releases>
|
121 |
|
|
<snapshots>
|
122 |
|
|
<enabled>false</enabled>
|
123 |
|
|
</snapshots>
|
124 |
|
|
<layout>default</layout>
|
125 |
|
|
</repository>
|
126 |
|
|
```
|
127 |
|
|
|
128 |
|
|
We also suggest to add the Tomcat plugin to the plugins group at the bottom of the same file:
|
129 |
|
|
|
130 |
|
|
```
|
131 |
|
|
<pluginGroups>
|
132 |
|
|
<pluginGroup>org.apache.tomcat.maven</pluginGroup>
|
133 |
|
|
</pluginGroups>
|
134 |
|
|
```
|
135 |
|
|
|
136 |
|
|
|
137 |
|
|
## Testing on local machine:
|
138 |
|
|
The D-Net Software is developed in Java with Maven. You can try out the D-Net web app on your local machine with the tomcat7 plugin.
|
139 |
|
|
```
|
140 |
|
|
> cd dnet-minimal-container
|
141 |
|
|
|
142 |
|
|
> mvn tomcat7:run
|
143 |
|
|
```
|
144 |
|
|
|
145 |
|
|
The webapp should be running at http://localhost:8280/app .
|
146 |
|
|
|
147 |
|
|
## Deployment on a Tomcat instance
|
148 |
|
|
The D-Net Software is developed in Java with Maven.
|
149 |
|
|
|
150 |
|
|
To build the war to use in a Tomcat 7 web app container:
|
151 |
|
|
|
152 |
|
|
```
|
153 |
|
|
> cd dnet-minimal-container
|
154 |
|
|
|
155 |
|
|
> mvn package
|
156 |
|
|
```
|
157 |
|
|
|
158 |
|
|
The .war file is then created into the <code>target</code> directory.
|
159 |
|
|
|
160 |
|
|
|