Index: modules/dnet-mapreduce-jobs/branches/yarn/deploy.info
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/deploy.info	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/deploy.info	(nonexistent)
@@ -1 +0,0 @@
-{"type_source": "SVN", "goal": "package -U -T 4C source:jar", "url": "http://svn-public.driver.research-infrastructures.eu/driver/dnet40/modules/dnet-mapreduce-jobs/trunk/", "deploy_repository": "dnet4-snapshots", "version": "4", "mail": "sandro.labruzzo@isti.cnr.it,michele.artini@isti.cnr.it, claudio.atzori@isti.cnr.it, alessia.bardi@isti.cnr.it", "deploy_repository_url": "http://maven.research-infrastructures.eu/nexus/content/repositories/dnet4-snapshots", "name": "dnet-mapreduce-jobs"}
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordWT.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordWT.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordWT.xml	(nonexistent)
@@ -1,44 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<record xmlns:dc="http://purl.org/dc/elements/1.1/"
-  xmlns:dr="http://www.driver-repository.eu/namespace/dr"
-  xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-  xmlns:oaf="http://namespace.openaire.eu/oaf"
-  xmlns:prov="http://www.openarchives.org/OAI/2.0/provenance" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
-  <header xmlns="http://namespace.openaire.eu/">
-    <dri:objIdentifier>od_______296::74732b873f881d9bff67c2a11b6a66d6</dri:objIdentifier>
-    <dri:recordIdentifier>ebfb74b7-3087-4da0-aaab-917121e40028_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==::oai:www.tdx.cat:10803/32188</dri:recordIdentifier>
-    <dri:dateOfCollection/>
-    <dri:mdFormat/>
-    <dri:mdFormatInterpretation/>
-    <dri:repositoryId>1d3fe9bd-a21d-4bba-9238-dc6259ee1424_UmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZXMvUmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZVR5cGU=</dri:repositoryId>
-    <dr:objectIdentifier/>
-    <dr:dateOfCollection>2014-07-12T06:03:38Z</dr:dateOfCollection>
-    <oaf:datasourceprefix>od_______296</oaf:datasourceprefix>
-  </header>
-  <metadata xmlns="http://namespace.openaire.eu/">
-    <dc:contributor>Riera Guilera, Carme</dc:contributor>
-    <dc:contributor>Universitat Autònoma de Barcelona. Departament de Filologia Espanyola</dc:contributor>
-    <dc:creator>Torras Francés, Meri</dc:creator>
-    <dc:dateAccepted>1998-10-26</dc:dateAccepted>
-    <dc:description>La tesi parteix d’una reflexió des de la teoria a propòsit de l’estatut de l’autobiografia (i del gènere epistolar en particular), preguntant-se a propòsit d’aspectes com són ara la construcció del subjecte en el discurs o el valor de l’experiència com a categoria de diferència. Per tal de aprofundir en aquesta aproximació, el segon capítol realitza un alectura crítica de dues cartes autobiogràfiques escrites per dones: la Carta a Sor Filotea de la Cruz, escrita per Sor Juana i l’ Autobiografía epistolar de Gertrudis Góme de Avellaneda. El capítol central analitza el binomi gènere femení-gènere epistolar per demostrar de quina manera va establir-se a la modernitat (la França dels segles XVII i XVIII) i per què la carta privada acaba definint-se com un gènere (genre) de gènere (gender) femení, sobre quins condicionaments socials va succeir i quin va ser el paper de les dones en tot això. En el darrer capítol s’aborda el seguiment del binomi a l’Espanya dels segles XVIII, XIX i XX, fent paleses les aplicacions particulars que va patir en aquest altre context historico-cultural. Tot això fent ús d’un plantejament i un enfocament propis de la filologia, la teoria de la literatura i la literatura comparada que, a l’apèndix, abraça altres arts, ja que s’hi recullen mostres de la pintura holandesa del xvii on el motiu de la carta femenina té un tractament plenament en consonància amb els paràmetres que van constituir l’assimmilació de gèneres (gènere femení-gènere epistolar) a la França dels salons.</dc:description>
-    <dc:description>La tesis parte de una reflexión desde la teoría a propósito del estatuto de la autobiografía (y del género epistolar en particular), preguntándose sobre aspectos como la construcción del sujeto en el discurso o el valor de la experiencia como categoría de diferencia. En aras de ahondar en esta aproximación, el segundo capítulo realiza una lectura crítica de dos cartas autobiográficas escritas por mujeres: la Carta a Sor Filotea de la Cruz, escrita por Sor Juana y la Autobiografía epistolar de Gertrudis Gómez de Avellaneda. El capítulo central está destinado a analizar el binomio género femenino-género epistolar para demostrar de qué modo se estableció en la modernidad (en la Francia de los siglos XVII y XVIII) y por qué la carta privada terminó definiéndose como un género (genre) de género (gender) femenino, sobre qué condicionamientos sociales aconteció y cuál fue el papel de las mujeres en ello. En el capítulo posterior se acomete el seguimiento del binomio en la España de los siglos XVIII, XIX y XX, dando cuenta de las aplicaciones particulares que sufrió en este otro contexto histórico-cultural. Todo ello haciendo uso de un planteamiento y un enfoque propios de la filología, la teoría de la literatura y la literatura comparada que, en el apéndice, abraza otras artes, puesto que en él se recogen muestras de la pintura holandesa del XVII donde el motivo de la carta femenina observa un tratamiento plenamente en consonancia con los parámetros que constituyeron la asimilación de los géneros (género femenino-género epistolar) en la Francia de los salones.</dc:description>
-    <dc:description>The thesis departs from a theoretic reflection on the status of autobiography in general —and of epistolary genre in particular— inquiring about aspects like the construction of the subject in the discourse or the value of experience as a category of difference. In order to examine in depth this approach, the second chapter performs a critical reading of two autobiographic letters written by women: the Carta a Sor Filotea de la Cruz by Sor Juana and the epistolary Autobiografía of Gertrudis Gómez de Avellaneda. The central chapter is devoted to analyze the epistolary genre—feminine gender relationship to show how it was established in Modernity (in France, XVII and XVIII centuries) and why the personal letter finished defined as a genre of feminine gender, on what social factors happened and what was the woman’s role in all that. The following chapter shows the relationship in Spain, XVIII, XIX and XX centuries, giving an account of particular applications that it suffered in this historical and cultural context. All this using approaches of Philology, Literary Theory and Comparative. The appendix considers some samples of the Dutch painting of the xvii where the motive of the feminine letter observes a treatment fully accordance with the parameters that constituted the assimilation of gender/genre (feminine gender/epistolary genre) in the French salons.</dc:description>
-    <dc:identifier>http://hdl.handle.net/10803/32188</dc:identifier>
-    <oaf:identifier identifierType="doi">10803/32188</oaf:identifier>
-    <oaf:identifier identifierType="orcidworkid">1234-1234-1234-1111</oaf:identifier>
-    <dc:language>esl/spa</dc:language>
-    <dc:publisher>Universitat Autònoma de Barcelona</dc:publisher>
-    <dc:title>La epistola privada como género: estrategias de construcción</dc:title>
-    <dc:source>TDX (Tesis Doctorals en Xarxa)</dc:source>
-    <dc:subject>Ciències Humanes</dc:subject>
-    <dc:subject>82 - Literatura</dc:subject>
-    <dr:CobjCategory>0006</dr:CobjCategory>
-    <dr:CobjCategory>0000</dr:CobjCategory>
-    <dr:CobjIdentifier>urn:isbn:9788469416310</dr:CobjIdentifier>
-    <oaf:collectedDatasourceid>opendoar____::296</oaf:collectedDatasourceid>
-    <oaf:accessrights/>
-    <oaf:projectid>wt__________::033742</oaf:projectid>
-    <oaf:hostedBy id="opendoar____::296" name="Tesis Doctorals en Xarxa"/>
-    <oaf:collectedFrom id="opendoar____::296" name="Tesis Doctorals en Xarxa"/>
-  </metadata>
-</record>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/project_organization.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/project_organization.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/project_organization.xml	(nonexistent)
@@ -1,16 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<ROW>
-    <FIELD name="resporganization">corda_______::937348734</FIELD>
-    <FIELD name="provenanceaction"
-        >sysimport:crosswalk:entityregistry@@@sysimport:crosswalk:entityregistry@@@dnet:provenanceActions@@@dnet:provenanceActions</FIELD>
-    <FIELD name="project">corda_______::295128</FIELD>
-    <FIELD name="startdate">2012-12-16</FIELD>
-    <FIELD name="participantnumber">1</FIELD>
-    <FIELD name="semantics"
-        >coordinator@@@coordinator@@@dnet:project_organization_relations@@@dnet:project_organization_relations</FIELD>
-    <FIELD name="trust">0.9</FIELD>
-    <FIELD name="inferenceprovenance" isNull="true"/>
-    <FIELD name="enddate">2017-12-15</FIELD>
-    <FIELD name="deletedbyinference">false</FIELD>
-    <FIELD name="inferred">false</FIELD>
-</ROW>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/datasource.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/datasource.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/datasource.xml	(nonexistent)
@@ -1,64 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<ROW>
-	<FIELD name="subjects">
-		<ITEM isNull="true" />
-	</FIELD>
-	<FIELD name="databaseaccessrestriction" isNull="true" />
-	<FIELD name="serviceprovider">false</FIELD>
-	<FIELD name="contactemail">agregator@ceon.pl</FIELD>
-	<FIELD name="collectedfromname">OpenAIRE</FIELD>
-	<FIELD name="datasourceid">openaire____::QWdyZWdhdG9yIENlT04=</FIELD>
-	<FIELD name="versioning">false</FIELD>
-	<FIELD name="deletedbyinference">false</FIELD>
-	<FIELD name="policies">
-		<ITEM isNull="true" />
-	</FIELD>
-	<FIELD name="odcontenttypes" />
-	<FIELD name="datauploadtype" isNull="true" />
-	<FIELD name="provenanceaction">user:insert@@@user:insert@@@dnet:provenanceActions@@@dnet:provenanceActions
-	</FIELD>
-	<FIELD name="dataprovider">false</FIELD>
-	<FIELD name="odnumberofitems" isNull="true" />
-	<FIELD name="namespaceprefix">agregator_ce</FIELD>
-	<FIELD name="description">Aggregator of Polish scientific publication
-		repositories, run by the ICM Centre for Open Science at the University
-		of Warsaw.</FIELD>
-	<FIELD name="citationguidelineurl" isNull="true" />
-	<FIELD name="odlanguages" />
-	<FIELD name="longitude">0.0</FIELD>
-	<FIELD name="pidsystems" isNull="true" />
-	<FIELD name="englishname">CeON Aggregator</FIELD>
-	<FIELD name="certificates" isNull="true" />
-	<FIELD name="releaseenddate" isNull="true" />
-	<FIELD name="odpolicies" isNull="true" />
-	<FIELD name="trust">0.9</FIELD>
-	<FIELD name="databaseaccesstype" isNull="true" />
-	<FIELD name="officialname">Agregator CeON</FIELD>
-	<FIELD name="pid">
-		<ITEM isNull="true" />
-	</FIELD>
-	<FIELD name="accessinfopackage">
-		<ITEM>http://agregator.ceon.pl/oai/oai.do</ITEM>
-		<ITEM isNull="true" />
-	</FIELD>
-	<FIELD name="inferred">false</FIELD>
-	<FIELD name="datasourcetype">aggregator::pubsrepository::unknown@@@Aggregator of
-		Publication
-		Repositories@@@dnet:datasource_typologies@@@dnet:datasource_typologies
-	</FIELD>
-	<FIELD name="collectedfromid">infrastruct_::openaire</FIELD>
-	<FIELD name="openairecompatibility">driver@@@OpenAIRE Basic (DRIVER
-		OA)@@@dnet:datasourceCompatibilityLevel@@@dnet:datasourceCompatibilityLevel
-	</FIELD>
-	<FIELD name="datauploadrestriction" isNull="true" />
-	<FIELD name="dateofvalidation">2014-11-24</FIELD>
-	<FIELD name="logourl"></FIELD>
-	<FIELD name="odnumberofitemsdate" isNull="true" />
-	<FIELD name="releasestartdate" isNull="true" />
-	<FIELD name="websiteurl">http://agregator.ceon.pl/</FIELD>
-	<FIELD name="inferenceprovenance" isNull="true" />
-	<FIELD name="qualitymanagementkind" isNull="true" />
-	<FIELD name="latitude">0.0</FIELD>
-	<FIELD name="dateofcollection">2014-11-24</FIELD>
-	<FIELD name="missionstatementurl" isNull="true" />
-</ROW>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordDatacite2.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordDatacite2.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordDatacite2.xml	(nonexistent)
@@ -1,59 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<record xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-	xmlns:oaf="http://namespace.openaire.eu/oaf"
-	xmlns:dr="http://www.driver-repository.eu/namespace/dr">
-	<oai:header xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-		xmlns:oai="http://www.openarchives.org/OAI/2.0/">
-		<dri:objIdentifier>datacite____::ebc6557dfca95e1180f3102b76e1ea50</dri:objIdentifier>
-		<dri:recordIdentifier>oai:oai.datacite.org:1500584</dri:recordIdentifier>
-		<dri:dateOfCollection>2014-09-14T16:12:21.24Z</dri:dateOfCollection>
-		<dri:repositoryId>1ab23232-a569-447f-8e74-5d3238e2e3c8_UmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZXMvUmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZVR5cGU=</dri:repositoryId>
-		<oaf:datasourceprefix>datacite____</oaf:datasourceprefix>
-		<identifier xmlns="http://www.openarchives.org/OAI/2.0/">oai:oai.datacite.org:1500584</identifier>
-		<datestamp xmlns="http://www.openarchives.org/OAI/2.0/">2014-07-03T21:52:38Z</datestamp>
-		<setSpec xmlns="http://www.openarchives.org/OAI/2.0/">CDL</setSpec>
-		<setSpec xmlns="http://www.openarchives.org/OAI/2.0/">CDL.DIGSCI</setSpec>
-	</oai:header>
-	<metadata>
-		<resource xmlns="http://datacite.org/schema/kernel-3"
-			xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-			xmlns:oai="http://www.openarchives.org/OAI/2.0/"
-			xsi:schemaLocation="http://datacite.org/schema/kernel-3 http://schema.datacite.org/meta/kernel-3/metadata.xsd">
-			<identifier identifierType="DOI">10.6084/M9.FIGSHARE.97674</identifier>
-			<titles>
-				<title>Increased beta-catenin protein and somatic APC mutations in sporadic aggressive
-					fibromatoses (desmoid tumors)</title>
-			</titles>
-			<publisher>Figshare</publisher>
-			<publicationYear>2012</publicationYear>
-			<resourceType resourceTypeGeneral="Dataset">Paper</resourceType>
-			<rightsList>
-				<rights rightsURI="http://creativecommons.org/licenses/by/3.0/us/">CC-BY</rights>
-			</rightsList>
-			<creators>
-				<creator>
-					<creatorName>Salvador J. Diaz-Cano</creatorName>
-					<nameIdentifier nameIdentifierScheme="ORCID">0000-0003-1245-2859</nameIdentifier>
-				</creator>
-				<creator>
-					<creatorName>B A Alman</creatorName>
-				</creator>
-				<creator>
-					<creatorName>C Li</creatorName>
-				</creator>
-				<creator>
-					<creatorName>H J Wolfe</creatorName>
-				</creator>
-				<creator>
-					<creatorName>M E Pajerski</creatorName>
-				</creator>
-			</creators>
-		</resource>
-		<dr:CobjCategory>0021</dr:CobjCategory>
-		<oaf:dateAccepted>2012-01-01</oaf:dateAccepted>
-		<oaf:accessrights>OPEN</oaf:accessrights>
-		<oaf:language>und</oaf:language>
-		<oaf:hostedBy name="Unknown Repository" id="openaire____::1256f046-bf1f-4afc-8b47-d0b147148b18"/>
-		<oaf:collectedFrom name="Datacite" id="openaire____::datacite"/>
-	</metadata>
-</record>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projectRecordFCT.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projectRecordFCT.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projectRecordFCT.xml	(nonexistent)
@@ -1,32 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<ROW>
-	<FIELD name="keywords">Chemistry and Biochemistry - Biochemistry</FIELD>
-	<FIELD name="acronym">UNKNOWN</FIELD>
-	<FIELD name="trust">0.9</FIELD>
-	<FIELD name="collectedfromname">FCT - Fundação para a Ciência e a Tecnologia</FIELD>
-	<FIELD name="pid">
-		<ITEM isNull="true" />
-	</FIELD>
-	<FIELD name="code">100007</FIELD>
-	<FIELD name="fundingtree">
-		<ITEM>&lt;fundingtree&gt;&lt;funder&gt;&lt;id&gt;fct_________::FCT&lt;/id&gt;&lt;shortname&gt;FCT&lt;/shortname&gt;&lt;name&gt;Funda&#231;&#227;o para a Ci&#234;ncia e a Tecnologia, I.P.&lt;/name&gt;&lt;/funder&gt;&lt;funding_level_1&gt;&lt;id&gt;fct_________::FCTFS::5876-PPCDTI&lt;/id&gt;&lt;description&gt;5876-PPCDTI&lt;/description&gt;&lt;name&gt;5876-PPCDTI&lt;/name&gt;&lt;class&gt;fct:program&lt;/class&gt;&lt;parent&gt;&lt;funding_level_0&gt;&lt;id&gt;fct_________::FCTFS&lt;/id&gt;&lt;description&gt;FCT Funding Stream&lt;/description&gt;&lt;name&gt;FCTFS&lt;/name&gt;&lt;parent/&gt;&lt;class&gt;fct:program&lt;/class&gt;&lt;/funding_level_0&gt;&lt;/parent&gt;&lt;/funding_level_1&gt;&lt;/fundingtree&gt;</ITEM>
-    </FIELD>
-	<FIELD name="deletedbyinference">false</FIELD>
-	<FIELD name="inferred">false</FIELD>
-	<FIELD name="collectedfromid">openaire____::fct</FIELD>
-	<FIELD name="ecsc39">false</FIELD>
-	<FIELD name="provenanceaction">sysimport:crosswalk:entityregistry@@@sysimport:crosswalk:entityregistry@@@dnet:provenanceActions@@@dnet:provenanceActions
-	</FIELD>
-	<FIELD name="title">Oxidative stress response mechanisms in Deinococcus radiodruans</FIELD>
-	<FIELD name="duration" isNull="true" />
-	<FIELD name="contracttype">UNKNOWN@@@UNKNOWN@@@fct:contractTypes@@@fct:contractTypes
-	</FIELD>
-	<FIELD name="startdate">2010-01-01</FIELD>
-	<FIELD name="websiteurl">http://www.fct.pt/apoios/projectos/consulta/vglobal_projecto.phtml.en?idProjecto=100007&amp;idElemConcurso=2700
-	</FIELD>
-	<FIELD name="inferenceprovenance" isNull="true" />
-	<FIELD name="callidentifier" isNull="true" />
-	<FIELD name="projectid">fct_________::100007</FIELD>
-	<FIELD name="enddate">2013-06-30</FIELD>
-	<FIELD name="dateofcollection">2014-12-12</FIELD>
-</ROW>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/organization.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/organization.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/organization.xml	(nonexistent)
@@ -1,31 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<ROW>
-    <FIELD name="echighereducation">false</FIELD>
-    <FIELD name="trust">0.9</FIELD>
-    <FIELD name="collectedfromname">CORDA - COmmon Research DAta Warehouse</FIELD>
-    <FIELD name="legalshortname">E.ON Global Commodities SE</FIELD>
-    <FIELD name="ecnonprofit">false</FIELD>
-    <FIELD name="pid">
-        <ITEM isNull="true"/>
-    </FIELD>
-    <FIELD name="ecsmevalidated" isNull="true"/>
-    <FIELD name="deletedbyinference">false</FIELD>
-    <FIELD name="inferred">false</FIELD>
-    <FIELD name="collectedfromid">openaire____::corda</FIELD>
-    <FIELD name="ecnutscode">false</FIELD>
-    <FIELD name="country">DE@@@Germany@@@dnet:countries@@@dnet:countries</FIELD>
-    <FIELD name="ecresearchorganization">false</FIELD>
-    <FIELD name="provenanceaction"
-        >sysimport:crosswalk:entityregistry@@@sysimport:crosswalk:entityregistry@@@dnet:provenanceActions@@@dnet:provenanceActions</FIELD>
-    <FIELD name="organizationid">corda_______::937348734</FIELD>
-    <FIELD name="ecinternationalorganizationeurinterests">false</FIELD>
-    <FIELD name="logourl" isNull="true"/>
-    <FIELD name="ecenterprise" isNull="true"/>
-    <FIELD name="legalname">E.ON GLOBAL COMMODITIES SE</FIELD>
-    <FIELD name="eclegalperson">true</FIELD>
-    <FIELD name="websiteurl">www.eon.com</FIELD>
-    <FIELD name="eclegalbody">false</FIELD>
-    <FIELD name="inferenceprovenance" isNull="true"/>
-    <FIELD name="dateofcollection">2014-09-22</FIELD>
-    <FIELD name="ecinternationalorganization">false</FIELD>
-</ROW>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/datasources_2_hbase.xsl
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/datasources_2_hbase.xsl	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/datasources_2_hbase.xsl	(nonexistent)
@@ -1,30 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<xsl:stylesheet version="1.0"
-	xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:dc="http://purl.org/dc/elements/1.1/"
-	xmlns:dr="http://www.driver-repository.eu/namespace/dr" xmlns:dri="http://www.driver-repository.eu/namespace/dri" 
-	xmlns:oaa="http://namespace.openaire.eu/oaa" xmlns:oaf="http://namespace.openaire.eu/oaf"
-	xmlns:dnet="eu.dnetlib.data.transform.xml.DNetDbToHbaseXsltFunctions">
-	
-	<xsl:output omit-xml-declaration="yes" indent="yes"/>
-
-	<xsl:template match="/">
-
-		<xsl:variable name="originalid" select="//FIELD[@name = 'datasourceid']" />
-		<xsl:variable name="datasourceid" select="dnet:oafSplitId('datasource', $originalid)" />		
-		<xsl:variable name="dateofcollection" select="//FIELD[@name = 'dateofcollection']" />
-		<xsl:variable name="collectedfromid" select="dnet:oafSplitId('datasource', //FIELD[@name = 'collectedfromid'])" />
-		<xsl:variable name="collectedfromname" select="//FIELD[@name = 'collectedfromname']" />
-		
-		<xsl:variable name="datasource" select="dnet:oafEntity('datasource', $datasourceid, $collectedfromid, $collectedfromname, $originalid, $dateofcollection, //FIELD[not(@isNull)])"/>
-		
-		<ROWS>
-			<xsl:if test="string-length($datasourceid) &gt; 0">
-				<ROW key="{$datasourceid}" columnFamily="datasource">
-					<QUALIFIER name="body" type="base64"><xsl:value-of select="$datasource"/></QUALIFIER>
-				</ROW>
-			</xsl:if>
-		</ROWS>
-	
-	</xsl:template>
-
-</xsl:stylesheet>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordFCT.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordFCT.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordFCT.xml	(nonexistent)
@@ -1,44 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<record xmlns:dc="http://purl.org/dc/elements/1.1/"
-  xmlns:dr="http://www.driver-repository.eu/namespace/dr"
-  xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-  xmlns:oaf="http://namespace.openaire.eu/oaf"
-  xmlns:prov="http://www.openarchives.org/OAI/2.0/provenance" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
-  <header xmlns="http://namespace.openaire.eu/">
-    <dri:objIdentifier>od_______296::74732b873f881d9bff67c2a11b6a66d6</dri:objIdentifier>
-    <dri:recordIdentifier>ebfb74b7-3087-4da0-aaab-917121e40028_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==::oai:www.tdx.cat:10803/32188</dri:recordIdentifier>
-    <dri:dateOfCollection/>
-    <dri:mdFormat/>
-    <dri:mdFormatInterpretation/>
-    <dri:repositoryId>1d3fe9bd-a21d-4bba-9238-dc6259ee1424_UmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZXMvUmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZVR5cGU=</dri:repositoryId>
-    <dr:objectIdentifier/>
-    <dr:dateOfCollection>2014-07-12T06:03:38Z</dr:dateOfCollection>
-    <oaf:datasourceprefix>od_______296</oaf:datasourceprefix>
-  </header>
-  <metadata xmlns="http://namespace.openaire.eu/">
-    <dc:contributor>Riera Guilera, Carme</dc:contributor>
-    <dc:contributor>Universitat Autònoma de Barcelona. Departament de Filologia Espanyola</dc:contributor>
-    <dc:creator>Torras Francés, Meri</dc:creator>
-    <dc:dateAccepted>1998-10-26</dc:dateAccepted>
-    <dc:description>La tesi parteix d’una reflexió des de la teoria a propòsit de l’estatut de l’autobiografia (i del gènere epistolar en particular), preguntant-se a propòsit d’aspectes com són ara la construcció del subjecte en el discurs o el valor de l’experiència com a categoria de diferència. Per tal de aprofundir en aquesta aproximació, el segon capítol realitza un alectura crítica de dues cartes autobiogràfiques escrites per dones: la Carta a Sor Filotea de la Cruz, escrita per Sor Juana i l’ Autobiografía epistolar de Gertrudis Góme de Avellaneda. El capítol central analitza el binomi gènere femení-gènere epistolar per demostrar de quina manera va establir-se a la modernitat (la França dels segles XVII i XVIII) i per què la carta privada acaba definint-se com un gènere (genre) de gènere (gender) femení, sobre quins condicionaments socials va succeir i quin va ser el paper de les dones en tot això. En el darrer capítol s’aborda el seguiment del binomi a l’Espanya dels segles XVIII, XIX i XX, fent paleses les aplicacions particulars que va patir en aquest altre context historico-cultural. Tot això fent ús d’un plantejament i un enfocament propis de la filologia, la teoria de la literatura i la literatura comparada que, a l’apèndix, abraça altres arts, ja que s’hi recullen mostres de la pintura holandesa del xvii on el motiu de la carta femenina té un tractament plenament en consonància amb els paràmetres que van constituir l’assimmilació de gèneres (gènere femení-gènere epistolar) a la França dels salons.</dc:description>
-    <dc:description>La tesis parte de una reflexión desde la teoría a propósito del estatuto de la autobiografía (y del género epistolar en particular), preguntándose sobre aspectos como la construcción del sujeto en el discurso o el valor de la experiencia como categoría de diferencia. En aras de ahondar en esta aproximación, el segundo capítulo realiza una lectura crítica de dos cartas autobiográficas escritas por mujeres: la Carta a Sor Filotea de la Cruz, escrita por Sor Juana y la Autobiografía epistolar de Gertrudis Gómez de Avellaneda. El capítulo central está destinado a analizar el binomio género femenino-género epistolar para demostrar de qué modo se estableció en la modernidad (en la Francia de los siglos XVII y XVIII) y por qué la carta privada terminó definiéndose como un género (genre) de género (gender) femenino, sobre qué condicionamientos sociales aconteció y cuál fue el papel de las mujeres en ello. En el capítulo posterior se acomete el seguimiento del binomio en la España de los siglos XVIII, XIX y XX, dando cuenta de las aplicaciones particulares que sufrió en este otro contexto histórico-cultural. Todo ello haciendo uso de un planteamiento y un enfoque propios de la filología, la teoría de la literatura y la literatura comparada que, en el apéndice, abraza otras artes, puesto que en él se recogen muestras de la pintura holandesa del XVII donde el motivo de la carta femenina observa un tratamiento plenamente en consonancia con los parámetros que constituyeron la asimilación de los géneros (género femenino-género epistolar) en la Francia de los salones.</dc:description>
-    <dc:description>The thesis departs from a theoretic reflection on the status of autobiography in general —and of epistolary genre in particular— inquiring about aspects like the construction of the subject in the discourse or the value of experience as a category of difference. In order to examine in depth this approach, the second chapter performs a critical reading of two autobiographic letters written by women: the Carta a Sor Filotea de la Cruz by Sor Juana and the epistolary Autobiografía of Gertrudis Gómez de Avellaneda. The central chapter is devoted to analyze the epistolary genre—feminine gender relationship to show how it was established in Modernity (in France, XVII and XVIII centuries) and why the personal letter finished defined as a genre of feminine gender, on what social factors happened and what was the woman’s role in all that. The following chapter shows the relationship in Spain, XVIII, XIX and XX centuries, giving an account of particular applications that it suffered in this historical and cultural context. All this using approaches of Philology, Literary Theory and Comparative. The appendix considers some samples of the Dutch painting of the xvii where the motive of the feminine letter observes a treatment fully accordance with the parameters that constituted the assimilation of gender/genre (feminine gender/epistolary genre) in the French salons.</dc:description>
-    <dc:identifier>http://hdl.handle.net/10803/32188</dc:identifier>
-    <oaf:identifier identifierType="doi">10803/32188</oaf:identifier>
-    <oaf:identifier identifierType="orcidworkid">1234-1234-1234-1111</oaf:identifier>
-    <dc:language>esl/spa</dc:language>
-    <dc:publisher>Universitat Autònoma de Barcelona</dc:publisher>
-    <dc:title>La epistola privada como género: estrategias de construcción</dc:title>
-    <dc:source>TDX (Tesis Doctorals en Xarxa)</dc:source>
-    <dc:subject>Ciències Humanes</dc:subject>
-    <dc:subject>82 - Literatura</dc:subject>
-    <dr:CobjCategory>0006</dr:CobjCategory>
-    <dr:CobjCategory>0000</dr:CobjCategory>
-    <dr:CobjIdentifier>urn:isbn:9788469416310</dr:CobjIdentifier>
-    <oaf:collectedDatasourceid>opendoar____::296</oaf:collectedDatasourceid>
-    <oaf:accessrights/>
-    <oaf:projectid>fct_________::100007</oaf:projectid>
-    <oaf:hostedBy id="opendoar____::296" name="Tesis Doctorals en Xarxa"/>
-    <oaf:collectedFrom id="opendoar____::296" name="Tesis Doctorals en Xarxa"/>
-  </metadata>
-</record>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordDatacite.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordDatacite.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordDatacite.xml	(nonexistent)
@@ -1,119 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<record xmlns:dr="http://www.driver-repository.eu/namespace/dr"
-  xmlns:oaf="http://namespace.openaire.eu/oaf" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
-  <oai:header xmlns="http://namespace.openaire.eu/"
-    xmlns:dc="http://purl.org/dc/elements/1.1/"
-    xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-    xmlns:oai="http://www.openarchives.org/OAI/2.0/" xmlns:prov="http://www.openarchives.org/OAI/2.0/provenance">
-    <dri:objIdentifier>r3c903d4c5a8::00a0cdbbc0c27cca1ccfafb33e7a862d</dri:objIdentifier>
-    <dri:recordIdentifier/>
-    <dri:dateOfCollection>2014-10-08T13:12:50.337Z</dri:dateOfCollection>
-    <dri:repositoryId>19365384-4f97-4446-9a81-8c414d2ca27a_UmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZXMvUmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZVR5cGU=</dri:repositoryId>
-    <oaf:datasourceprefix>r3c903d4c5a8</oaf:datasourceprefix>
-    <identifier xmlns="http://www.openarchives.org/OAI/2.0/">oai:ucd.ie:ucdlib:31727</identifier>
-    <datestamp xmlns="http://www.openarchives.org/OAI/2.0/">2014-10-07T14:28:12Z</datestamp>
-    <setSpec xmlns="http://www.openarchives.org/OAI/2.0/">ucd_digital_library</setSpec>
-    <setSpec xmlns="http://www.openarchives.org/OAI/2.0/">openaire_data</setSpec>
-  </oai:header>
-  <metadata>
-    <resource xmlns="http://datacite.org/schema/kernel-3"
-      xmlns:dc="http://purl.org/dc/elements/1.1/"
-      xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-      xmlns:oai="http://www.openarchives.org/OAI/2.0/"
-      xmlns:prov="http://www.openarchives.org/OAI/2.0/provenance" xsi:schemaLocation="http://datacite.org/schema/kernel-3 http://schema.datacite.org/meta/kernel-3/metadata.xsd">
-      <identifier identifierType="DOI">10.7925/drs1.ucdlib_31727</identifier>
-      <creators>
-        <creator>
-          <creatorName>Laefer, Debra L.</creatorName>
-          <nameIdentifier nameIdentifierScheme="ORCID">0000-0001-5134-5322</nameIdentifier>
-        </creator>
-        <creator>
-          <creatorName>University College, Dublin. Library</creatorName>
-          <nameIdentifier nameIdentifierScheme="VIAF" schemeURI="http://www.viaf.org">261450561</nameIdentifier>
-        </creator>
-      </creators>
-      <titles>
-        <title>39 Grafton Street, Dublin</title>
-      </titles>
-      <publisher>UCD School of Architecture Landscape Civil Engineering</publisher>
-      <publicationYear>2013</publicationYear>
-      <subjects>
-        <subject
-          schemeURI="http://id.loc.gov/authorities/subjects/sh99004820#concept" subjectScheme="NAF">Buildings, structures, etc</subject>
-      </subjects>
-      <contributors>
-        <contributor contributorType="Other">
-          <contributorName>Clarke, Julie</contributorName>
-          <nameIdentifier nameIdentifierScheme="ORCID">0000-0001-6092-9741</nameIdentifier>
-        </contributor>
-        <contributor contributorType="Sponsor">
-          <contributorName>UCD School of Architecture, Landscape, and Civil Engineering</contributorName>
-        </contributor>
-        <contributor contributorType="HostingInstitution">
-          <contributorName>University College, Dublin. Library</contributorName>
-          <nameIdentifier nameIdentifierScheme="VIAF" schemeURI="http://www.viaf.org">261450561</nameIdentifier>
-        </contributor>
-        <contributor contributorType="Funder">
-          <contributorName>Irish Research Council for Science, Engineering and Technology</contributorName>
-          <nameIdentifier nameIdentifierScheme="info">info:eu-repo/grantAgreement/IRCSET//RS-2011-435</nameIdentifier>
-        </contributor>
-      </contributors>
-      <dates>
-        <date dateType="Issued">2013</date>
-        <date dateType="Available">2014-09-29</date>
-      </dates>
-      <resourceType resourceTypeGeneral="Dataset">Dataset</resourceType>
-      <alternateIdentifiers>
-        <alternateIdentifier alternateIdentifierType="URL">http://digital.ucd.ie/view/ucdlib:31727</alternateIdentifier>
-      </alternateIdentifiers>
-      <relatedIdentifiers>
-        <relatedIdentifier relatedIdentifierType="DOI" relationType="IsPartOf">10.7925/drs1.ucdlib_31100</relatedIdentifier>
-        <relatedIdentifier relatedIdentifierType="DOI" relationType="IsReferencedBy">10.7925/drs1.ucdlib_31057</relatedIdentifier>
-      </relatedIdentifiers>
-      <sizes>
-        <size>1 still image</size>
-        <size>26859085 bytes</size>
-        <size>1545x2946 pixels</size>
-      </sizes>
-      <formats>
-        <format>image/vnd.adobe.photoshop</format>
-        <format>image/jpeg</format>
-      </formats>
-      <rightsList>
-        <rights rightsURI="info:eu-repo/semantics/openAccess"/>
-        <rights rightsURI="http://creativecommons.org/publicdomain/zero/1.0/">Use and reproduction: CC0 1.0 Universal</rights>
-      </rightsList>
-      <descriptions>
-        <description descriptionType="Abstract">Consists of an orthorectified image of the structure in Adobe PhotoShop (PSD) format. The orthorectified image has been derived from one or more photographic images that are included as layers in the original Adobe PhotoShop (PSD) file. The composite and all other image layers have been extracted and saved in TIFF image files, and JPEG files in various resolutions have been derived from the flattened image layer to facilitate inspection.</description>
-        <description descriptionType="Other">Brickwork structure, 4 levels above ground, constructed 1864.</description>
-        <description descriptionType="Other">Funded by an award from the Irish Research Council for Science, Engineering and Technology (IRCSET), Embark Initiative Postgraduate Scholarship, grant identifier RS/2011/435.</description>
-      </descriptions>
-      <geoLocations>
-        <geoLocation>
-          <geoLocationPoint>53.3407951 -6.26054559383235</geoLocationPoint>
-        </geoLocation>
-      </geoLocations>
-    </resource>
-    <oaf:embargoenddate>2014-09-29</oaf:embargoenddate>
-    <dr:CobjCategory>0021</dr:CobjCategory>
-    <oaf:dateAccepted>2013-01-01</oaf:dateAccepted>
-    <oaf:accessrights>OPEN</oaf:accessrights>
-    <oaf:language>und</oaf:language>
-    <oaf:hostedBy id="re3data_____::r3d100010742" name="UCD Digital Library"/>
-    <oaf:collectedFrom id="re3data_____::r3d100010742" name="UCD Digital Library"/>
-  </metadata>
-  <about xmlns="http://namespace.openaire.eu/"
-    xmlns:dc="http://purl.org/dc/elements/1.1/"
-    xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-    xmlns:oai="http://www.openarchives.org/OAI/2.0/" xmlns:prov="http://www.openarchives.org/OAI/2.0/provenance">
-    <oaf:datainfo>
-      <oaf:inferred>false</oaf:inferred>
-      <oaf:deletedbyinference>false</oaf:deletedbyinference>
-      <oaf:trust>0.9</oaf:trust>
-      <oaf:inferenceprovenance/>
-      <oaf:provenanceaction classid="sysimport:crosswalk:datasetarchive"
-        classname="sysimport:crosswalk:datasetarchive"
-        schemeid="dnet:provenanceActions" schemename="dnet:provenanceActions"/>
-    </oaf:datainfo>
-  </about>
-</record>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/doajUniversityRecord.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/doajUniversityRecord.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/doajUniversityRecord.xml	(nonexistent)
@@ -1,87 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<record xmlns:dc="http://purl.org/dc/elements/1.1/"
-    xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-    xmlns:prov="http://www.openarchives.org/OAI/2.0/provenance"
-    xmlns:oaf="http://namespace.openaire.eu/oaf"
-    xmlns:dr="http://www.driver-repository.eu/namespace/dr">
-    <header xmlns="http://namespace.openaire.eu/">
-        <dri:objIdentifier>doajarticles::00c586857161af61d02d7e2b2754e0d6</dri:objIdentifier>
-        <dri:recordIdentifier>oai:doaj.org/article:ca57383f9d4742bcaaed3e067a95bebc</dri:recordIdentifier>
-        <dri:dateOfCollection/>
-        <dri:mdFormat/>
-        <dri:mdFormatInterpretation/>
-        <dri:repositoryId>fe9d4996-f5cb-4dac-a47d-f6cfaf8a0fcc_UmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZXMvUmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZVR5cGU=</dri:repositoryId>
-        <dr:objectIdentifier/>
-        <dr:dateOfCollection>2014-12-01T09:12:32Z</dr:dateOfCollection>
-        <oaf:datasourceprefix>doajarticles</oaf:datasourceprefix>
-    </header>
-    <metadata xmlns="http://namespace.openaire.eu/">
-        <dc:title>Management issues power quality in distribution system</dc:title>
-        <dc:creator>Daniel Bălan</dc:creator>
-        <dc:date>2011-12-01T00:00:00Z</dc:date>
-        <dc:description>In recent years, the number of power quality problems has increased
-            dramatically. This increase in electricity demand of recent years, consecutively with
-            increasing the number of power plants and power, along with changing the structure of
-            energy agents and increasepower receivers, led to deterioration of power quality
-            parameters, with implications serious about working arrangements and consumer networks.
-            Most disputed aspect in terms of power quality standards is the regime's control sizes.
-            A clear understanding of issues related to power quality management in all its
-            complexity, correctly identifying the causes, phenomena and present solutions in the
-            area of power quality can help to avoid damage / failure, that the increased
-            benefit</dc:description>
-        <dc:identifier>http://scientificbulletin.upm.ro/papers/2011/12/Balan-Daniel-Management-issues-power-quality-in-distribution-system2.pdf</dc:identifier>
-        <dc:language>eng</dc:language>
-        <dc:publisher>Editura Universităţii "Petru Maior"</dc:publisher>
-        <dc:source>Scientific Bulletin of the ''Petru Maior" University of Tîrgu Mureș, Vol 8 (XXV),
-            Iss 2, Pp 70-77 (2011)</dc:source>
-        <dc:subject>LCC:Engineering (General). Civil engineering (General)</dc:subject>
-        <dc:subject>LCC:TA1-2040</dc:subject>
-        <dc:subject>LCC:Technology</dc:subject>
-        <dc:subject>LCC:T</dc:subject>
-        <dc:subject>DOAJ:General and Civil Engineering</dc:subject>
-        <dc:subject>DOAJ:Technology and Engineering</dc:subject>
-        <dc:subject>LCC:Engineering (General). Civil engineering (General)</dc:subject>
-        <dc:subject>LCC:TA1-2040</dc:subject>
-        <dc:subject>LCC:Technology</dc:subject>
-        <dc:subject>LCC:T</dc:subject>
-        <dc:subject>DOAJ:General and Civil Engineering</dc:subject>
-        <dc:subject>DOAJ:Technology and Engineering</dc:subject>
-        <dc:subject>LCC:Engineering (General). Civil engineering (General)</dc:subject>
-        <dc:subject>LCC:TA1-2040</dc:subject>
-        <dc:subject>LCC:Technology</dc:subject>
-        <dc:subject>LCC:T</dc:subject>
-        <dc:subject>LCC:Engineering (General). Civil engineering (General)</dc:subject>
-        <dc:subject>LCC:TA1-2040</dc:subject>
-        <dc:subject>LCC:Technology</dc:subject>
-        <dc:subject>LCC:T</dc:subject>
-        <dc:subject>LCC:Engineering (General). Civil engineering (General)</dc:subject>
-        <dc:subject>LCC:TA1-2040</dc:subject>
-        <dc:subject>LCC:Technology</dc:subject>
-        <dc:subject>LCC:T</dc:subject>
-        <dc:type/>
-        <dr:CobjCategory>0001</dr:CobjCategory>
-        <oaf:dateAccepted>2011-12-01</oaf:dateAccepted>
-        <oaf:projectid/>
-        <oaf:collectedDatasourceid>driver______::1790119e-d281-4b7a-aedf-866d1d853a07</oaf:collectedDatasourceid>
-        <oaf:accessrights>OPEN</oaf:accessrights>
-        <oaf:hostedBy
-            name="Scientific Bulletin of the ''Petru Maior&quot; University of Tîrgu Mureș"
-            id="doajarticles::1841-9267"/>
-        <oaf:collectedFrom name="DOAJ-Articles"
-            id="driver______::1790119e-d281-4b7a-aedf-866d1d853a07"/>
-        <oaf:journal issn="1841-9267" eissn="">Scientific Bulletin of the ''Petru Maior" University
-            of Tîrgu Mureș</oaf:journal>
-    </metadata>
-    <about xmlns="http://namespace.openaire.eu/" xmlns:oai="http://www.openarchives.org/OAI/2.0/">
-        <oaf:datainfo>
-            <oaf:inferred>false</oaf:inferred>
-            <oaf:deletedbyinference>false</oaf:deletedbyinference>
-            <oaf:trust>0.9</oaf:trust>
-            <oaf:inferenceprovenance/>
-            <oaf:provenanceaction schemename="dnet:provenanceActions"
-                schemeid="dnet:provenanceActions" classname="sysimport:crosswalk:repository"
-                classid="sysimport:crosswalk:repository"/>
-        </oaf:datainfo>
-    </about>
-</record>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projectRecordCorda.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projectRecordCorda.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projectRecordCorda.xml	(nonexistent)
@@ -1,32 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<ROW>
-	<FIELD name="keywords" isNull="true"/>
-	<FIELD name="acronym">NANOBRIDGES</FIELD>
-	<FIELD name="trust">0.9</FIELD>
-	<FIELD name="collectedfromname">CORDA - COmmon Research DAta Warehouse</FIELD>
-	<FIELD name="pid">
-		<ITEM isNull="true"/>
-	</FIELD>
-	<FIELD name="code">295128</FIELD>
-	<FIELD name="fundingtree">
-		<ITEM>&lt;fundingtree&gt;&lt;funder&gt;&lt;id&gt;ec__________::EC&lt;/id&gt;&lt;shortname&gt;EC&lt;/shortname&gt;&lt;name&gt;European Commission&lt;/name&gt;&lt;/funder&gt;&lt;funding_level_2&gt;&lt;id&gt;corda_______::FP7::SP4::SME&lt;/id&gt;&lt;description&gt;Research for the benefit of SMEs&lt;/description&gt;&lt;name&gt;SME&lt;/name&gt;&lt;class&gt;ec:program&lt;/class&gt;&lt;parent&gt;&lt;funding_level_1&gt;&lt;id&gt;corda_______::FP7::SP4&lt;/id&gt;&lt;description&gt;SP4-Capacities&lt;/description&gt;&lt;name&gt;SP4&lt;/name&gt;&lt;class&gt;ec:specificprogram&lt;/class&gt;&lt;parent&gt;&lt;funding_level_0&gt;&lt;id&gt;corda_______::FP7&lt;/id&gt;&lt;description&gt;SEVENTH FRAMEWORK PROGRAMME&lt;/description&gt;&lt;name&gt;FP7&lt;/name&gt;&lt;parent/&gt;&lt;class&gt;ec:frameworkprogram&lt;/class&gt;&lt;/funding_level_0&gt;&lt;/parent&gt;&lt;/funding_level_1&gt;&lt;/parent&gt;&lt;/funding_level_2&gt;&lt;/fundingtree&gt;</ITEM>
-	</FIELD>
-	<FIELD name="deletedbyinference">false</FIELD>
-	<FIELD name="inferred">false</FIELD>
-	<FIELD name="collectedfromid">openaire____::corda</FIELD>
-	<FIELD name="ecsc39">false</FIELD>
-	<FIELD name="provenanceaction"
-		>sysimport:crosswalk:entityregistry@@@sysimport:crosswalk:entityregistry@@@dnet:provenanceActions@@@dnet:provenanceActions</FIELD>
-	<FIELD name="title">Building bridges between specialists on computational and empirical risk
-		assessment of engineered nanomaterials</FIELD>
-	<FIELD name="duration" isNull="true"/>
-	<FIELD name="contracttype">MC@@@Support for training and career development of researchers (Marie
-		Curie)@@@ec:FP7contractTypes@@@ec:FP7contractTypes</FIELD>
-	<FIELD name="startdate">2012-01-01</FIELD>
-	<FIELD name="websiteurl" isNull="true"/>
-	<FIELD name="inferenceprovenance" isNull="true"/>
-	<FIELD name="callidentifier">FP7-PEOPLE-2011-IRSES</FIELD>
-	<FIELD name="projectid">corda_______::295128</FIELD>
-	<FIELD name="enddate">2014-12-31</FIELD>
-	<FIELD name="dateofcollection">2013-05-07</FIELD>
-</ROW>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/record.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/record.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/record.xml	(nonexistent)
@@ -1,50 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<record xmlns:dc="http://purl.org/dc/elements/1.1/"
-  xmlns:dr="http://www.driver-repository.eu/namespace/dr"
-  xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-  xmlns:oaf="http://namespace.openaire.eu/oaf"
-  xmlns:prov="http://www.openarchives.org/OAI/2.0/provenance" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
-  <header xmlns="http://namespace.openaire.eu/">
-    <dri:objIdentifier>od_______296::74732b873f881d9bff67c2a11b6a66d6</dri:objIdentifier>
-    <dri:recordIdentifier>ebfb74b7-3087-4da0-aaab-917121e40028_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==::oai:www.tdx.cat:10803/32188</dri:recordIdentifier>
-    <dri:dateOfCollection/>
-    <dri:mdFormat/>
-    <dri:mdFormatInterpretation/>
-    <dri:repositoryId>1d3fe9bd-a21d-4bba-9238-dc6259ee1424_UmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZXMvUmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZVR5cGU=</dri:repositoryId>
-    <dr:objectIdentifier/>
-    <dr:dateOfCollection>2014-07-12T06:03:38Z</dr:dateOfCollection>
-    <oaf:datasourceprefix>od_______296</oaf:datasourceprefix>
-  </header>
-  <metadata xmlns="http://namespace.openaire.eu/">
-    <dc:contributor>Riera Guilera, Carme</dc:contributor>
-    <dc:contributor>Universitat Autònoma de Barcelona. Departament de Filologia Espanyola</dc:contributor>
-    <dc:creator>Torras Francés, Meri</dc:creator>
-    <dc:creator>Moreno-Noguer, Francese</dc:creator>
-    <dc:dateAccepted>1998-10-26</dc:dateAccepted>
-    <dc:description>La tesi parteix d’una reflexió des de la teoria a propòsit de l’estatut de l’autobiografia (i del gènere epistolar en particular), preguntant-se a propòsit d’aspectes com són ara la construcció del subjecte en el discurs o el valor de l’experiència com a categoria de diferència. Per tal de aprofundir en aquesta aproximació, el segon capítol realitza un alectura crítica de dues cartes autobiogràfiques escrites per dones: la Carta a Sor Filotea de la Cruz, escrita per Sor Juana i l’ Autobiografía epistolar de Gertrudis Góme de Avellaneda. El capítol central analitza el binomi gènere femení-gènere epistolar per demostrar de quina manera va establir-se a la modernitat (la França dels segles XVII i XVIII) i per què la carta privada acaba definint-se com un gènere (genre) de gènere (gender) femení, sobre quins condicionaments socials va succeir i quin va ser el paper de les dones en tot això. En el darrer capítol s’aborda el seguiment del binomi a l’Espanya dels segles XVIII, XIX i XX, fent paleses les aplicacions particulars que va patir en aquest altre context historico-cultural. Tot això fent ús d’un plantejament i un enfocament propis de la filologia, la teoria de la literatura i la literatura comparada que, a l’apèndix, abraça altres arts, ja que s’hi recullen mostres de la pintura holandesa del xvii on el motiu de la carta femenina té un tractament plenament en consonància amb els paràmetres que van constituir l’assimmilació de gèneres (gènere femení-gènere epistolar) a la França dels salons.</dc:description>
-    <dc:description>La tesis parte de una reflexión desde la teoría a propósito del estatuto de la autobiografía (y del género epistolar en particular), preguntándose sobre aspectos como la construcción del sujeto en el discurso o el valor de la experiencia como categoría de diferencia. En aras de ahondar en esta aproximación, el segundo capítulo realiza una lectura crítica de dos cartas autobiográficas escritas por mujeres: la Carta a Sor Filotea de la Cruz, escrita por Sor Juana y la Autobiografía epistolar de Gertrudis Gómez de Avellaneda. El capítulo central está destinado a analizar el binomio género femenino-género epistolar para demostrar de qué modo se estableció en la modernidad (en la Francia de los siglos XVII y XVIII) y por qué la carta privada terminó definiéndose como un género (genre) de género (gender) femenino, sobre qué condicionamientos sociales aconteció y cuál fue el papel de las mujeres en ello. En el capítulo posterior se acomete el seguimiento del binomio en la España de los siglos XVIII, XIX y XX, dando cuenta de las aplicaciones particulares que sufrió en este otro contexto histórico-cultural. Todo ello haciendo uso de un planteamiento y un enfoque propios de la filología, la teoría de la literatura y la literatura comparada que, en el apéndice, abraza otras artes, puesto que en él se recogen muestras de la pintura holandesa del XVII donde el motivo de la carta femenina observa un tratamiento plenamente en consonancia con los parámetros que constituyeron la asimilación de los géneros (género femenino-género epistolar) en la Francia de los salones.</dc:description>
-    <dc:description>The thesis departs from a theoretic reflection on the status of autobiography in general —and of epistolary genre in particular— inquiring about aspects like the construction of the subject in the discourse or the value of experience as a category of difference. In order to examine in depth this approach, the second chapter performs a critical reading of two autobiographic letters written by women: the Carta a Sor Filotea de la Cruz by Sor Juana and the epistolary Autobiografía of Gertrudis Gómez de Avellaneda. The central chapter is devoted to analyze the epistolary genre—feminine gender relationship to show how it was established in Modernity (in France, XVII and XVIII centuries) and why the personal letter finished defined as a genre of feminine gender, on what social factors happened and what was the woman’s role in all that. The following chapter shows the relationship in Spain, XVIII, XIX and XX centuries, giving an account of particular applications that it suffered in this historical and cultural context. All this using approaches of Philology, Literary Theory and Comparative. The appendix considers some samples of the Dutch painting of the xvii where the motive of the feminine letter observes a treatment fully accordance with the parameters that constituted the assimilation of gender/genre (feminine gender/epistolary genre) in the French salons.</dc:description>
-    <dc:identifier>http://hdl.handle.net/10803/32188</dc:identifier>
-    <oaf:identifier identifierType="doi">10803/32188</oaf:identifier>
-    <oaf:identifier identifierType="orcidworkid">1234-1234-1234-1111</oaf:identifier>
-    <dc:language>esl/spa</dc:language>
-    <dc:publisher>Universitat Autònoma de Barcelona</dc:publisher>
-    <dc:title>La epistola privada como género: estrategias de construcción</dc:title>
-    <dc:source>TDX (Tesis Doctorals en Xarxa)</dc:source>
-    <dc:subject>Ciències Humanes</dc:subject>
-    <dc:subject>82 - Literatura</dc:subject>
-    <dr:CobjCategory>0006</dr:CobjCategory>
-    <dr:CobjCategory>0000</dr:CobjCategory>
-    <dr:CobjIdentifier>urn:isbn:9788469416310</dr:CobjIdentifier>
-    <oaf:collectedDatasourceid>opendoar____::296</oaf:collectedDatasourceid>
-    <oaf:accessrights/>
-    <oaf:reference 
-    	identifier="CHEMBL1658379" 
-    	query="http://www.ebi.ac.uk/europepmc/webservices/rest/PMC/PMC2934972/databaseLinks/ChEMBL/2/json" 
-    	source="CHEMBL" 
-    	title="Antifungal activity against Candida albicans isolate 108 harboring erg11 and erg5 double mutant after 48 hrs by Etest" 
-    	type="dataset">https://www.ebi.ac.uk/chembldb/assay/inspect/CHEMBL1658379</oaf:reference>
-    <oaf:hostedBy id="opendoar____::296" name="Tesis Doctorals en Xarxa"/>
-    <oaf:collectedFrom id="opendoar____::296" name="Tesis Doctorals en Xarxa"/>
-  </metadata>
-</record>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/mdstore_cleaned.json.gz
===================================================================
Cannot display: file marked as a binary type.
svn:mime-type = application/octet-stream
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/mdstore_cleaned.json.gz
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/mdstore_cleaned.json.gz	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/mdstore_cleaned.json.gz	(nonexistent)

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/mdstore_cleaned.json.gz
___________________________________________________________________
Deleted: svn:mime-type
## -1 +0,0 ##
-application/octet-stream
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/organizations_2_hbase.xsl
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/organizations_2_hbase.xsl	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/organizations_2_hbase.xsl	(nonexistent)
@@ -1,30 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<xsl:stylesheet version="1.0"
-	xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:dc="http://purl.org/dc/elements/1.1/"
-	xmlns:dr="http://www.driver-repository.eu/namespace/dr" xmlns:dri="http://www.driver-repository.eu/namespace/dri" 
-	xmlns:oaa="http://namespace.openaire.eu/oaa" xmlns:oaf="http://namespace.openaire.eu/oaf"
-	xmlns:dnet="eu.dnetlib.data.transform.xml.DNetDbToHbaseXsltFunctions">
-	
-	<xsl:output omit-xml-declaration="yes" indent="yes"/>
-
-	<xsl:template match="/">
-		
-		<xsl:variable name="originalid" select="//FIELD[@name = 'organizationid']" />
-		<xsl:variable name="organizationid" select="dnet:oafSplitId('organization', $originalid)" />
-		<xsl:variable name="dateofcollection" select="//FIELD[@name = 'dateofcollection']" />
-		<xsl:variable name="collectedfromid" select="dnet:oafSplitId('datasource', //FIELD[@name = 'collectedfromid'])" />
-		<xsl:variable name="collectedfromname" select="//FIELD[@name = 'collectedfromname']" />		
-
-		<xsl:variable name="organization" select="dnet:oafEntity('organization', $organizationid, $collectedfromid, $collectedfromname, $originalid, $dateofcollection, //FIELD[not(@isNull)])"/>
-		
-		<ROWS>
-			<xsl:if test="string-length($organizationid) &gt; 0">
-				<ROW key="{$organizationid}" columnFamily="organization">
-					<QUALIFIER name="body" type="base64"><xsl:value-of select="$organization"/></QUALIFIER>
-				</ROW>
-			</xsl:if>
-		</ROWS>
-	
-	</xsl:template>
-
-</xsl:stylesheet>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projectRecordWT.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projectRecordWT.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projectRecordWT.xml	(nonexistent)
@@ -1,54 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<ROW>
-	<FIELD name="keywords">Senior Research Fellowships In Clinical Science
-	</FIELD>
-	<FIELD name="acronym"></FIELD>
-	<FIELD name="trust">0.9</FIELD>
-	<FIELD name="collectedfromname">UK Pubmed Central - Wellcome Trust</FIELD>
-	<FIELD name="pid">
-		<ITEM isNull="true" />
-	</FIELD>
-	<FIELD name="code">033742</FIELD>
-	<FIELD name="fundingtree">
-		&lt;fundingtree&gt;
-		&lt;funder&gt;
-		&lt;id&gt;wt__________::WT&lt;/id&gt;
-		&lt;shortname&gt;WT&lt;/shortname&gt;
-		&lt;name&gt;Wellcome Trust&lt;/name&gt;
-		&lt;/funder&gt;
-		&lt;funding_level_1&gt;
-		&lt;id&gt;wt__________::WT::Physiological Sciences&lt;/id&gt;
-		&lt;description&gt;Physiological Sciences&lt;/description&gt;
-		&lt;name&gt;Physiological Sciences&lt;/name&gt;
-		&lt;class&gt;wt:fundingStream&lt;/class&gt;
-		&lt;parent&gt;
-		&lt;funding_level_0&gt;
-		&lt;id&gt;wt__________::WT&lt;/id&gt;
-		&lt;description&gt;Wellcome Trust Funding Stream&lt;/description&gt;
-		&lt;name&gt;WT&lt;/name&gt;
-		&lt;parent /&gt;
-		&lt;class&gt;wt:fundingStream&lt;/class&gt;
-		&lt;/funding_level_0&gt;
-		&lt;/parent&gt;
-		&lt;/funding_level_1&gt;
-		&lt;/fundingtree&gt; </FIELD>
-	<FIELD name="deletedbyinference">false</FIELD>
-	<FIELD name="inferred">false</FIELD>
-	<FIELD name="collectedfromid">openaire____::wellcometrust</FIELD>
-	<FIELD name="ecsc39">false</FIELD>
-	<FIELD name="provenanceaction">sysimport:crosswalk:entityregistry@@@sysimport:crosswalk:entityregistry@@@dnet:provenanceActions@@@dnet:provenanceActions
-	</FIELD>
-	<FIELD name="title">Insulin resistance: causes and consequences
-		(renewal).
-	</FIELD>
-	<FIELD name="duration" isNull="true" />
-	<FIELD name="contracttype">UNKNOWN@@@UNKNOWN@@@wt:contractTypes@@@wt:contractTypes
-	</FIELD>
-	<FIELD name="startdate">1996-08-01</FIELD>
-	<FIELD name="websiteurl" isNull="true" />
-	<FIELD name="inferenceprovenance" isNull="true" />
-	<FIELD name="callidentifier" isNull="true" />
-	<FIELD name="projectid">wt__________::033742</FIELD>
-	<FIELD name="enddate">2005-09-30</FIELD>
-	<FIELD name="dateofcollection">2013-05-07</FIELD>
-</ROW>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/pangaeOAF.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/pangaeOAF.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/pangaeOAF.xml	(nonexistent)
@@ -1,29 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<oai:record xmlns:dc="http://purl.org/dc/elements/1.1/"
-  xmlns:dr="http://www.driver-repository.eu/namespace/dr"
-  xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-  xmlns:md="http://www.pangaea.de/MetaData"
-  xmlns:oaf="http://namespace.openaire.eu/oaf" xmlns:oai="http://www.openarchives.org/OAI/2.0/">
-  <oai:header>
-    <dri:objIdentifier>r39633d1e8c4::00848fca3eb9c38b0f7895e959a2b3b8</dri:objIdentifier>
-    <dri:recordIdentifier>ref20384</dri:recordIdentifier>
-    <dri:dateOfCollection>2014-10-30T10:13:35.945Z</dri:dateOfCollection>
-    <dri:repositoryId/>
-    <oaf:datasourceprefix>r39633d1e8c4</oaf:datasourceprefix>
-  </oai:header>
-  <oai:metadata>
-    <dc:identifier>ref20384</dc:identifier>
-    <dc:title>The oxygen and carbon isotope composition of Quaternary bivalve shells as a water mass indicator: last interglacial and Holocene, East Greenland</dc:title>
-    <dc:creator>Israelson C</dc:creator>
-    <dc:creator>Buchardt B</dc:creator>
-    <dc:creator>Funder Svend</dc:creator>
-    <dc:creator>Hubberten Hans-Wolfgang</dc:creator>
-    <dc:source>Palaeogeography, Palaeoclimatology, Palaeoecology</dc:source>
-    <oaf:journal>Palaeogeography, Palaeoclimatology, Palaeoecology</oaf:journal>
-    <oaf:projectid>corda_______::295128</oaf:projectid>
-    <oaf:hostedBy
-      id="openaire____::1256f046-bf1f-4afc-8b47-d0b147148b18" name="Unknown Repository"/>
-    <oaf:collectedFrom id="re3data_____::r3d100010134" name="PANGAEA"/>
-    <oaf:relatedDataset id="r39633d1e8c4::00a1866ddd695c6400a91e04e9d23ffe"/>
-  </oai:metadata>
-</oai:record>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projects_2_hbase.xsl
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projects_2_hbase.xsl	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projects_2_hbase.xsl	(nonexistent)
@@ -1,30 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<xsl:stylesheet version="1.0"
-	xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:dc="http://purl.org/dc/elements/1.1/"
-	xmlns:dr="http://www.driver-repository.eu/namespace/dr" xmlns:dri="http://www.driver-repository.eu/namespace/dri" 
-	xmlns:oaa="http://namespace.openaire.eu/oaa" xmlns:oaf="http://namespace.openaire.eu/oaf"
-	xmlns:dnet="eu.dnetlib.data.transform.xml.DNetDbToHbaseXsltFunctions">
-	
-	<xsl:output omit-xml-declaration="yes" indent="yes"/>
-
-	<xsl:template match="/">
-	
-		<xsl:variable name="originalid" select="//FIELD[@name = 'projectid']" />
-		<xsl:variable name="projectid" select="dnet:oafSplitId('project', $originalid)" />
-		<xsl:variable name="dateofcollection" select="//FIELD[@name = 'dateofcollection']" />		
-		<xsl:variable name="collectedfromid" select="dnet:oafSplitId('datasource', //FIELD[@name = 'collectedfromid'])" />
-		<xsl:variable name="collectedfromname" select="//FIELD[@name = 'collectedfromname']" />			
-
-		<xsl:variable name="project" select="dnet:oafEntity('project', $projectid, $collectedfromid, $collectedfromname, $originalid, $dateofcollection, //FIELD[not(@isNull)])"/>
-
-		<ROWS>
-			<xsl:if test="string-length($projectid) &gt; 0">
-				<ROW key="{$projectid}" columnFamily="project">
-					<QUALIFIER name="body" type="base64"><xsl:value-of select="$project"/></QUALIFIER>
-				</ROW>
-			</xsl:if>
-		</ROWS>
-	
-	</xsl:template>
-
-</xsl:stylesheet>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projectorganization_2_hbase.xsl
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projectorganization_2_hbase.xsl	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/projectorganization_2_hbase.xsl	(nonexistent)
@@ -1,34 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<xsl:stylesheet version="1.0"
-	xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:dc="http://purl.org/dc/elements/1.1/"
-	xmlns:dr="http://www.driver-repository.eu/namespace/dr" xmlns:dri="http://www.driver-repository.eu/namespace/dri" 
-	xmlns:oaa="http://namespace.openaire.eu/oaa" xmlns:oaf="http://namespace.openaire.eu/oaf"
-	xmlns:dnet="eu.dnetlib.data.transform.xml.DNetDbToHbaseXsltFunctions">
-	
-	<xsl:output omit-xml-declaration="yes" indent="yes"/>
-
-	<xsl:template match="/">
-	
-		<xsl:variable name="projectId" select="dnet:oafSplitId('project', //FIELD[@name = 'project'])" />
-		<xsl:variable name="organizationId" select="dnet:oafSplitId('organization', //FIELD[@name = 'resporganization'])" />
-
-		<ROWS>
-			<xsl:if test="string-length($projectId) &gt; 0 and string-length($organizationId) &gt; 0">
-			
-				<xsl:variable name="projectorganization" 
-						select="dnet:oafRel('projectOrganization', $projectId, $organizationId, //FIELD[not(@isNull)], 'hasParticipant', 'dnet:project_organization_relations')"/>
-				<xsl:variable name="organizationproject" 
-						select="dnet:oafRel('projectOrganization', $organizationId, $projectId, //FIELD[not(@isNull)], 'isParticipant', 'dnet:project_organization_relations')"/>	
-			
-				<ROW key="{$projectId}" columnFamily="projectOrganization_participation_hasParticipant">
-					<QUALIFIER name="{$organizationId}" type="base64"><xsl:value-of select="$projectorganization"/></QUALIFIER>
-				</ROW>
-				<ROW key="{$organizationId}" columnFamily="projectOrganization_participation_isParticipant">
-					<QUALIFIER name="{$projectId}" type="base64"><xsl:value-of select="$organizationproject"/></QUALIFIER>
-				</ROW>
-			</xsl:if>
-		</ROWS>
-	
-	</xsl:template>
-
-</xsl:stylesheet>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/pangaeODF.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/pangaeODF.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/pangaeODF.xml	(nonexistent)
@@ -1,42 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<oai:record xmlns:dc="http://purl.org/dc/elements/1.1/"
-  xmlns:dr="http://www.driver-repository.eu/namespace/dr"
-  xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-  xmlns:md="http://www.pangaea.de/MetaData"
-  xmlns:oaf="http://namespace.openaire.eu/oaf" xmlns:oai="http://www.openarchives.org/OAI/2.0/">
-  <oai:header>
-    <dri:objIdentifier>r39633d1e8c4::00a1866ddd695c6400a91e04e9d23ffe</dri:objIdentifier>
-    <dri:recordIdentifier>10.1594/PANGAEA.55062</dri:recordIdentifier>
-    <dri:dateOfCollection>2014-10-30T10:12:53.842Z</dri:dateOfCollection>
-    <dri:repositoryId/>
-    <oaf:datasourceprefix>r39633d1e8c4</oaf:datasourceprefix>
-  </oai:header>
-  <oai:metadata>
-    <resource xmlns="http://datacite.org/schema/kernel-3"
-      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://datacite.org/schema/kernel-3 http://schema.datacite.org/meta/kernel-3/metadata.xsd">
-      <identifier identifierType="DOI">10.1594/PANGAEA.55062</identifier>
-      <creators>
-        <creator>
-          <creatorName>Norbert R Nowaczyk</creatorName>
-          <nameIdentifier nameIdentifierScheme="PANGAEA" schemeURI="http://www.pangaea.de">dataset.author398</nameIdentifier>
-        </creator>
-        <creator>
-          <creatorName>Marlyse Baumann</creatorName>
-          <nameIdentifier nameIdentifierScheme="PANGAEA" schemeURI="http://www.pangaea.de">dataset.author10442</nameIdentifier>
-        </creator>
-      </creators>
-      <titles>
-        <title xml:lang="en-us">Coccoliths of sediment core PS1519-12</title>
-      </titles>
-      <publisher>PANGAEA</publisher>
-      <publicationYear>1999</publicationYear>
-      <descriptions>
-        <description descriptionType="Abstract"/>
-      </descriptions>
-      <oaf:projectid>corda_______::304178</oaf:projectid>
-      <oaf:hostedBy id="re3data_____::r3d100010134" name="PANGAEA"/>
-      <oaf:collectedFrom id="re3data_____::r3d100010134" name="PANGAEA"/>
-      <oaf:relatedPublication id="r39633d1e8c4::00848fca3eb9c38b0f7895e959a2b3b8"/>
-    </resource>
-  </oai:metadata>
-</oai:record>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordClaim.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordClaim.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordClaim.xml	(nonexistent)
@@ -1,42 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<record xmlns:dc="http://purl.org/dc/elements/1.1/"
-  xmlns:dr="http://www.driver-repository.eu/namespace/dr"
-  xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-  xmlns:oaf="http://namespace.openaire.eu/oaf"
-  xmlns:prov="http://www.openarchives.org/OAI/2.0/provenance" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
-  <header xmlns="http://namespace.openaire.eu/">
-    <dri:objIdentifier>crossref____::0018a1d45d8ea09a68824dc12dd56064</dri:objIdentifier>
-    <dri:recordIdentifier>10.1109/TSP.2011.2140106</dri:recordIdentifier>
-    <dri:dateOfCollection/>
-    <dri:mdFormat/>
-    <dri:mdFormatInterpretation/>
-    <dri:repositoryId/>
-    <dr:objectIdentifier/>
-    <dr:dateOfCollection>2014-04-14T18:02:25+03:00</dr:dateOfCollection>
-    <oaf:datasourceprefix>crossref____</oaf:datasourceprefix>
-  </header>
-  <metadata xmlns="http://namespace.openaire.eu/">
-    <dc:dateAccepted>2011-01-01-01-01T00:00:00Z</dc:dateAccepted>
-    <dc:identifier>http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5744133</dc:identifier>
-    <dc:title>Sparse Variational Bayesian SAGE Algorithm With Application to the Estimation of Multipath Wireless Channels</dc:title>
-    <oaf:collectedDatasourceid>openaire____::crossref</oaf:collectedDatasourceid>
-    <oaf:accessrights>RESTRICTED</oaf:accessrights>
-    <oaf:hostedBy
-      id="openaire____::1256f046-bf1f-4afc-8b47-d0b147148b18" name="Unknown Repository"/>
-    <oaf:collectedFrom id="openaire____::crossref" name="Crossref"/>
-    <dc:creator>Shutin, D.</dc:creator>
-    <dc:creator>Fleury, B. H.</dc:creator>
-    <dc:identifier identifierType="doi">10.1109/TSP.2011.2140106</dc:identifier>
-    <dr:CobjIdentifier>10.1109/TSP.2011.2140106</dr:CobjIdentifier>
-  </metadata>
-  <about>
-    <oaf:datainfo>
-      <oaf:inferred>false</oaf:inferred>
-      <oaf:deletedbyinference>false</oaf:deletedbyinference>
-      <oaf:trust>0.92</oaf:trust>
-      <oaf:inferenceprovenance/>
-      <oaf:provenanceaction classid="user:claim:pid"
-        classname="user:claim:pid" schemeid="dnet:provenanceActions" schemename="dnet:provenanceActions"/>
-    </oaf:datainfo>  
-  </about>
-</record>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordCorda.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordCorda.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordCorda.xml	(nonexistent)
@@ -1,41 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<record xmlns:dc="http://purl.org/dc/elements/1.1/"
-  xmlns:dr="http://www.driver-repository.eu/namespace/dr"
-  xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-  xmlns:oaf="http://namespace.openaire.eu/oaf"
-  xmlns:prov="http://www.openarchives.org/OAI/2.0/provenance" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
-  <header xmlns="http://namespace.openaire.eu/">
-    <dri:objIdentifier>od_______296::74732b873f881d9bff67c2a11b6a66d6</dri:objIdentifier>
-    <dri:recordIdentifier>ebfb74b7-3087-4da0-aaab-917121e40028_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==::oai:www.tdx.cat:10803/32188</dri:recordIdentifier>
-    <dri:dateOfCollection/>
-    <dri:mdFormat/>
-    <dri:mdFormatInterpretation/>
-    <dri:repositoryId>1d3fe9bd-a21d-4bba-9238-dc6259ee1424_UmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZXMvUmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZVR5cGU=</dri:repositoryId>
-    <dr:objectIdentifier/>
-    <dr:dateOfCollection>2014-07-12T06:03:38Z</dr:dateOfCollection>
-    <oaf:datasourceprefix>od_______296</oaf:datasourceprefix>
-  </header>
-  <metadata xmlns="http://namespace.openaire.eu/">
-    <dc:dateAccepted>1998-10-26</dc:dateAccepted>
-    <dc:description>La tesi parteix d’una reflexió des de la teoria a propòsit de l’estatut de l’autobiografia (i del gènere epistolar en particular), preguntant-se a propòsit d’aspectes com són ara la construcció del subjecte en el discurs o el valor de l’experiència com a categoria de diferència. Per tal de aprofundir en aquesta aproximació, el segon capítol realitza un alectura crítica de dues cartes autobiogràfiques escrites per dones: la Carta a Sor Filotea de la Cruz, escrita per Sor Juana i l’ Autobiografía epistolar de Gertrudis Góme de Avellaneda. El capítol central analitza el binomi gènere femení-gènere epistolar per demostrar de quina manera va establir-se a la modernitat (la França dels segles XVII i XVIII) i per què la carta privada acaba definint-se com un gènere (genre) de gènere (gender) femení, sobre quins condicionaments socials va succeir i quin va ser el paper de les dones en tot això. En el darrer capítol s’aborda el seguiment del binomi a l’Espanya dels segles XVIII, XIX i XX, fent paleses les aplicacions particulars que va patir en aquest altre context historico-cultural. Tot això fent ús d’un plantejament i un enfocament propis de la filologia, la teoria de la literatura i la literatura comparada que, a l’apèndix, abraça altres arts, ja que s’hi recullen mostres de la pintura holandesa del xvii on el motiu de la carta femenina té un tractament plenament en consonància amb els paràmetres que van constituir l’assimmilació de gèneres (gènere femení-gènere epistolar) a la França dels salons.</dc:description>
-    <dc:description>La tesis parte de una reflexión desde la teoría a propósito del estatuto de la autobiografía (y del género epistolar en particular), preguntándose sobre aspectos como la construcción del sujeto en el discurso o el valor de la experiencia como categoría de diferencia. En aras de ahondar en esta aproximación, el segundo capítulo realiza una lectura crítica de dos cartas autobiográficas escritas por mujeres: la Carta a Sor Filotea de la Cruz, escrita por Sor Juana y la Autobiografía epistolar de Gertrudis Gómez de Avellaneda. El capítulo central está destinado a analizar el binomio género femenino-género epistolar para demostrar de qué modo se estableció en la modernidad (en la Francia de los siglos XVII y XVIII) y por qué la carta privada terminó definiéndose como un género (genre) de género (gender) femenino, sobre qué condicionamientos sociales aconteció y cuál fue el papel de las mujeres en ello. En el capítulo posterior se acomete el seguimiento del binomio en la España de los siglos XVIII, XIX y XX, dando cuenta de las aplicaciones particulares que sufrió en este otro contexto histórico-cultural. Todo ello haciendo uso de un planteamiento y un enfoque propios de la filología, la teoría de la literatura y la literatura comparada que, en el apéndice, abraza otras artes, puesto que en él se recogen muestras de la pintura holandesa del XVII donde el motivo de la carta femenina observa un tratamiento plenamente en consonancia con los parámetros que constituyeron la asimilación de los géneros (género femenino-género epistolar) en la Francia de los salones.</dc:description>
-    <dc:description>The thesis departs from a theoretic reflection on the status of autobiography in general —and of epistolary genre in particular— inquiring about aspects like the construction of the subject in the discourse or the value of experience as a category of difference. In order to examine in depth this approach, the second chapter performs a critical reading of two autobiographic letters written by women: the Carta a Sor Filotea de la Cruz by Sor Juana and the epistolary Autobiografía of Gertrudis Gómez de Avellaneda. The central chapter is devoted to analyze the epistolary genre—feminine gender relationship to show how it was established in Modernity (in France, XVII and XVIII centuries) and why the personal letter finished defined as a genre of feminine gender, on what social factors happened and what was the woman’s role in all that. The following chapter shows the relationship in Spain, XVIII, XIX and XX centuries, giving an account of particular applications that it suffered in this historical and cultural context. All this using approaches of Philology, Literary Theory and Comparative. The appendix considers some samples of the Dutch painting of the xvii where the motive of the feminine letter observes a treatment fully accordance with the parameters that constituted the assimilation of gender/genre (feminine gender/epistolary genre) in the French salons.</dc:description>
-    <dc:identifier>http://hdl.handle.net/10803/32188</dc:identifier>
-    <oaf:identifier identifierType="doi">10803/32188</oaf:identifier>
-    <oaf:identifier identifierType="orcidworkid">1234-1234-1234-1111</oaf:identifier>
-    <dc:language>esl/spa</dc:language>
-    <dc:publisher>Universitat Autònoma de Barcelona</dc:publisher>
-    <dc:title>La epistola privada como género: estrategias de construcción</dc:title>
-    <dc:source>TDX (Tesis Doctorals en Xarxa)</dc:source>
-    <dc:subject>Ciències Humanes</dc:subject>
-    <dc:subject>82 - Literatura</dc:subject>
-    <dr:CobjCategory>0006</dr:CobjCategory>
-    <dr:CobjCategory>0000</dr:CobjCategory>
-    <dr:CobjIdentifier>urn:isbn:9788469416310</dr:CobjIdentifier>
-    <oaf:collectedDatasourceid>opendoar____::296</oaf:collectedDatasourceid>
-    <oaf:accessrights/>
-    <oaf:projectid>corda_______::295128</oaf:projectid>
-    <oaf:hostedBy id="opendoar____::296" name="Tesis Doctorals en Xarxa"/>
-    <oaf:collectedFrom id="opendoar____::296" name="Tesis Doctorals en Xarxa"/>
-  </metadata>
-</record>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordArxiv.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordArxiv.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/recordArxiv.xml	(nonexistent)
@@ -1,53 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<record xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:prov="http://www.openarchives.org/OAI/2.0/provenance" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:dri="http://www.driver-repository.eu/namespace/dri" xmlns:dr="http://www.driver-repository.eu/namespace/dr" xmlns:oaf="http://namespace.openaire.eu/oaf">
-  <header xmlns="http://namespace.openaire.eu/">
-    <dri:objIdentifier>od________18::0000246aff5cef3992efe4a18a4b4972</dri:objIdentifier>
-    <dri:recordIdentifier>a49ee326-3d66-4f95-b775-362c7be0af2a_TURTdG9yZURTUmVzb3VyY2VzL01EU3RvcmVEU1Jlc291cmNlVHlwZQ==::oai:arXiv.org:1003.3012</dri:recordIdentifier>
-    <dri:dateOfCollection/>
-    <dri:mdFormat/>
-    <dri:mdFormatInterpretation/>
-    <dri:repositoryId>feaee13a-e5fb-4b5b-b1ba-9c27aa2ecc92_UmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZXMvUmVwb3NpdG9yeVNlcnZpY2VSZXNvdXJjZVR5cGU=</dri:repositoryId>
-    <dr:objectIdentifier/>
-    <dr:dateOfCollection>2014-01-28T13:52:45Z</dr:dateOfCollection>
-    <oaf:datasourceprefix>od________18</oaf:datasourceprefix>
-  </header>
-  <metadata xmlns="http://namespace.openaire.eu/">
-    <dc:creator>Bousso, Raphael</dc:creator>
-    <dc:creator>Freivogel, Ben</dc:creator>
-    <dc:creator>Leichenauer, Stefan</dc:creator>
-    <dc:dateAccepted>2010-03-15</dc:dateAccepted>
-    <dc:description>  The covariant entropy bound states that the entropy, S, of matter on a
-light-sheet cannot exceed a quarter of its initial area, A, in Planck units.
-The gravitational entropy of black holes saturates this inequality. The entropy
-of matter systems, however, falls short of saturating the bound in known
-examples. This puzzling gap has led to speculation that a much stronger bound,
-S&lt; A^{3/4}, may hold true. In this note, we exhibit light-sheets whose entropy
-exceeds A^{3/4} by arbitrarily large factors. In open FRW universes, such
-light-sheets contain the entropy visible in the sky; in the limit of early
-curvature domination, the covariant bound can be saturated but not violated. As
-a corollary, we find that the maximum observable matter and radiation entropy
-in universes with positive (negative) cosmological constant is of order
-Lambda^{-1} (Lambda^{-2}), and not |Lambda|^{-3/4} as had hitherto been
-believed. Our results strengthen the evidence for the covariant entropy bound,
-while showing that the stronger bound S&lt; A^{3/4} is not universally valid. We
-conjecture that the stronger bound does hold for static, weakly gravitating
-systems.
-</dc:description>
-    <dc:description>Comment: 36 pages, 10 figures</dc:description>
-    <dc:identifier>http://arxiv.org/abs/1003.3012</dc:identifier>
-    <dc:language>eng</dc:language>
-    <dc:title>Saturating the holographic entropy bound</dc:title>
-    <dc:subject>High Energy Physics - Theory</dc:subject>
-    <dc:subject>Astrophysics - Cosmology and Extragalactic Astrophysics</dc:subject>
-    <dc:subject>General Relativity and Quantum Cosmology</dc:subject>
-    <dc:subject>High Energy Physics - Phenomenology</dc:subject>
-    <dr:CobjCategory>0001</dr:CobjCategory>
-    <dr:CobjIdentifier>Phys.Rev.D82:084024,2010</dr:CobjIdentifier>
-    <dr:CobjIdentifier>doi:10.1103/PhysRevD.82.084024</dr:CobjIdentifier>
-    <oaf:collectedDatasourceid>opendoar____::18</oaf:collectedDatasourceid>
-    <oaf:accessrights>OPEN</oaf:accessrights>
-    <oaf:hostedBy name="arXiv.org e-Print Archive" id="opendoar____::18"/>
-    <oaf:collectedFrom name="arXiv.org e-Print Archive" id="opendoar____::18"/>
-    <oaf:identifier identifierType="doi">10.1103/PhysRevD.82.084024</oaf:identifier>
-  </metadata>
-</record>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/simpleRecord.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/simpleRecord.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/simpleRecord.xml	(nonexistent)
@@ -1,7 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<record>
-  <metadata>
-    <a>A text value</a>
-    <b attr="attribute value"/>
-  </metadata>
-</record>
\ No newline at end of file

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/transform/simpleRecord.xml
___________________________________________________________________
Deleted: svn:mime-type
## -1 +0,0 ##
-text/plain
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/mapreduce/util/openaireEntityLinks.properties
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/mapreduce/util/openaireEntityLinks.properties	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/mapreduce/util/openaireEntityLinks.properties	(nonexistent)
@@ -1,6 +0,0 @@
-#Rels
-result=resultOrganization:organization:rel,personResult:person:rel,instance:result:child
-person=personResult:result:rel
-datasource=datasourceOrganization:organization:rel
-organization=projectOrganization:project:rel,resultOrganization:result:rel
-project=projectOrganization:organization:rel,resultProject:result:rel
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/mapreduce/dedup/titles.txt
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/mapreduce/dedup/titles.txt	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/mapreduce/dedup/titles.txt	(nonexistent)
@@ -1,15 +0,0 @@
-ZZZZZZZZ
-AAAAAAAAAAAAAAAAA
-CCCCCCC
-CCCCCCC
-XXXXXXXXXX
-AAAAAAAAAAAAAAAAA
-AAAAAAAAAAAAAAAAA
-AAAAAAAAAAAAAAAAA
-BBBBBBBBBBBB
-AAAAAAAAAAAAAAAAA
-CCCCCCC
-AAAAAAAAAAAAAAAAA
-CCCCCCC
-CCCCCCC
-CCCCCCC
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIPublisherConfiguration-1.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIPublisherConfiguration-1.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIPublisherConfiguration-1.xml	(nonexistent)
@@ -1,59 +0,0 @@
-<RESOURCE_PROFILE>
-	<HEADER>
-		<RESOURCE_IDENTIFIER value="OAIPublisherConfiguration" />
-		<RESOURCE_TYPE value="OAIPublisherConfigurationDSResourceType" />
-		<RESOURCE_KIND value="OAIPublisherConfigurationDSResources" />
-		<RESOURCE_URI value="" />
-		<DATE_OF_CREATION value="2001-12-31T12:00:00" />
-	</HEADER>
-	<BODY>
-		<CONFIGURATION>
-            <IDSCHEME>oai</IDSCHEME>
-            <IDNAMESPACE>dnet</IDNAMESPACE>
-			<OAISETS>
-				<OAISET enabled="true">
-					<spec>set1</spec>
-					<name>set number 1</name>
-					<description>test set</description>
-					<query>(set ="aSet")</query>
-				</OAISET>
-			</OAISETS>
-			<METADATAFORMATS>
-				<METADATAFORMAT exportable="true" metadataPrefix="oai_dc">
-					<NAMESPACE>NAMESPACE</NAMESPACE>
-					<SCHEMA>SCHEMA</SCHEMA>
-					<SOURCE_METADATA_FORMAT interpretation="openaire" layout="index" name="oaf" />
-					<TRANSFORMATION_RULE />
-					<BASE_QUERY />
-				</METADATAFORMAT>
-				<METADATAFORMAT exportable="true" metadataPrefix="hope">
-					<NAMESPACE>NAMESPACE</NAMESPACE>
-					<SCHEMA>SCHEMA</SCHEMA>
-					<SOURCE_METADATA_FORMAT interpretation="I" layout="L" name="hope" />
-					<TRANSFORMATION_RULE>tdsRule</TRANSFORMATION_RULE>
-					<BASE_QUERY>country = IT</BASE_QUERY>
-				</METADATAFORMAT>
-			</METADATAFORMATS>
-			<INDICES>
-				<INDEX name="title">
-					<SOURCE interpretation="I" layout="L" name="hope" path="//title" />
-					<SOURCE interpretation="I" layout="L" name="hope" path="//titleinfo" />
-					<SOURCE interpretation="openaire" layout="index" name="oaf" path="//*[local-name() = 'title']" />
-					<SOURCE interpretation="driver" layout="store" name="oai_dc" path="//*[local-name() ='alternative']" />
-				</INDEX>
-				<INDEX name="objIdentifier">
-					<SOURCE interpretation="openaire" layout="index" name="oaf" path="//*[local-name() ='objIdentifier']" />
-					<SOURCE interpretation="driver" layout="store" name="oai_dc" path="//*[local-name() ='objIdentifier']" />
-				</INDEX>
-				<INDEX name="set">
-					<SOURCE interpretation="openaire" layout="index" name="oaf" path="//*[local-name() ='repositoryId']" />
-					<SOURCE interpretation="driver" layout="store" name="oai_dc" path="//*[local-name() ='repositoryId']" />
-				</INDEX>
-			</INDICES>
-		</CONFIGURATION>
-		<STATUS>
-			<LAST_UPDATE value="2013-12-20T10:50:00" />
-		</STATUS>
-		<SECURITY_PARAMETERS>SECURITY_PARAMETERS</SECURITY_PARAMETERS>
-	</BODY>
-</RESOURCE_PROFILE>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/mapreduce/hbase/oai/dedupedRecord.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/mapreduce/hbase/oai/dedupedRecord.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/resources/eu/dnetlib/data/mapreduce/hbase/oai/dedupedRecord.xml	(nonexistent)
@@ -1,391 +0,0 @@
-
-<record>
-	<result xmlns:dri="http://www.driver-repository.eu/namespace/dri"
-		xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
-		<header>
-			<dri:objIdentifier>wodc______nl::81199feaafe60a7ea6c9eb9b2b9ffd3a</dri:objIdentifier>
-			<dri:dateOfCollection>2013-12-11T10:01:13Z</dri:dateOfCollection>
-		</header>
-		<metadata>
-			<oaf:entity xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-				xmlns:oaf="http://namespace.openaire.eu/oaf"
-				xsi:schemaLocation="http://namespace.openaire.eu/oaf http://www.openaire.eu/schema/0.2/oaf-0.2.xsd">
-				<oaf:result>
-					<subject classid="keyword" classname="keyword" schemeid="dnet:result_subject"
-						schemename="dnet:result_subject">Asielbeleid</subject>
-					<subject classid="keyword" classname="keyword" schemeid="dnet:result_subject"
-						schemename="dnet:result_subject">Vreemdelingenbeleid</subject>
-					<subject classid="keyword" classname="keyword" schemeid="dnet:result_subject"
-						schemename="dnet:result_subject">Vreemdelingenbewaring</subject>
-					<subject classid="keyword" classname="keyword" schemeid="dnet:result_subject"
-						schemename="dnet:result_subject">Vreemdelingentoezicht</subject>
-					<title classid="main title" classname="main title" schemeid="dnet:dataCite_title"
-						schemename="dnet:dataCite_title">Vreemdelingen in Nederland:</title>
-					<dateofacceptance>1986-01-01</dateofacceptance>
-					<resulttype classid="publication" classname="publication"
-						schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-					<language classid="und" classname="Undetermined" schemeid="dnet:languages"
-						schemename="dnet:languages" />
-					<relevantdate classid="" classname="" schemeid=""
-						schemename="" />
-					<description></description>
-					<publisher></publisher>
-					<embargoenddate></embargoenddate>
-					<storagedate></storagedate>
-					<source></source>
-					<fulltext></fulltext>
-					<resourcetype classid="" classname="" schemeid=""
-						schemename="" />
-					<device></device>
-					<size></size>
-					<format></format>
-					<version></version>
-					<lastmetadataupdate></lastmetadataupdate>
-					<metadataversionnumber></metadataversionnumber>
-					<originalId>oai:tudelft.nl:uuid:017d6abc-dc6c-4560-b3bc-16bab777978e
-					</originalId>
-					<collectedfrom
-						name="WODC Repository Ministerie van Veiligheid en Justitie" id="driver______::7d8b320d781e5eced039bb46e6ec3c56" />
-					<pid classid="oai" classname="oai" schemeid="dnet:pid_types"
-						schemename="dnet:pid_types">oai:tudelft.nl:uuid:017d6abc-dc6c-4560-b3bc-16bab777978e
-					</pid>
-					<bestlicense classid="OPEN" classname="Open Access"
-						schemeid="dnet:access_modes" schemename="dnet:access_modes" />
-					<datainfo>
-						<inferred>false</inferred>
-						<deletedbyinference>true</deletedbyinference>
-						<trust>0.9</trust>
-						<inferenceprovenance />
-						<provenanceaction classid="sysimport:crosswalk:repository"
-							classname="sysimport:crosswalk:repository" schemeid="dnet:provenanceActions"
-							schemename="dnet:provenanceActions" />
-					</datainfo>
-					<rels>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">od_______105::2911ccfbfaca2d0fe8c365d46cf57779</to>
-							<similarity>0.39992455</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Geen onderwijs voor
-								vreemdelingen</title>
-							<dateofacceptance>2013-01-01</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">od_______202::35b9e745bd72719733bb59768033fea6</to>
-							<similarity>0.39992455</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Daardoor blijven wij
-								vreemdelingen</title>
-							<dateofacceptance>1996-09-14</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">doajarticles::0d7b9dfc3beb2556358a05ed2d7969cf</to>
-							<similarity>0.39992455</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Vreemdelingen?</title>
-							<dateofacceptance>1921-01-01</dateofacceptance>
-							<publisher>KITLV</publisher>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="false" trust="0.9" inferenceprovenance=""
-							provenanceaction="sysimport:crosswalk:repository">
-							<to class="hasAuthor" scheme="dnet:personroles" type="person">wodc______nl::1d713022bd10a53c5968df3203e33e16
-							</to>
-							<ranking>1</ranking>
-							<fullname>WODC</fullname>
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">od______1113::4c6e651be7385b0628fd82d5544586fc</to>
-							<similarity>0.39974904</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">De uitvoering van het
-								Nederlandse vreemdelingenbeleid</title>
-							<dateofacceptance>2006-01-11</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">wodc______nl::3a0f85380fbb663ebab94cea85bfa305</to>
-							<similarity>0.4196265</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Vreemdelingen:</title>
-							<dateofacceptance>1992-01-01</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">wodc______nl::8f24c874935af6d255faf2b8116a0901</to>
-							<similarity>0.5458018</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Operationeel
-								vreemdelingentoezicht in Nederland:</title>
-							<dateofacceptance>1989-01-01</dateofacceptance>
-							<publisher>Den Haag: Gouda Quint</publisher>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">wodc______nl::f9ee52cd455f4ef0fd6c9a9a73220a53</to>
-							<similarity>0.36704296</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Asiel zoeken in
-								Nederland:</title>
-							<dateofacceptance>1987-01-01</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">wodc______nl::d0267196f65eed0aa7a7e9326310e286</to>
-							<similarity>0.36079064</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Vreemdelingentoezicht
-								sinds de inwerkingtreding van de Wet op de Identificatieplicht:
-								Een verkennend onderzoek naar staande houden op grond van het
-								gewijzigde artikel 19 lid 1 Vreemdelingenwet</title>
-							<dateofacceptance>1997-01-01</dateofacceptance>
-							<publisher>Den Haag: WODC</publisher>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">wodc______nl::534e84e38868c13baed3d9cc3a2c1cc3</to>
-							<similarity>0.46265823</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Van bejegening tot
-								vertrek: Een onderzoek naar de werking van vreemdelingenbewaring
-							</title>
-							<dateofacceptance>2013-01-01</dateofacceptance>
-							<publisher>Den Haag: Boom Lemma</publisher>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">wodc______nl::a4e4adda854964e735d49f068f3625cd</to>
-							<similarity>0.6035621</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Operationeel toezicht
-								vreemdelingen: Evaluatie van de bevoegdheden in de
-								Vreemdelingenwet 2000 voor het vreemdelingentoezicht door de
-								politie</title>
-							<dateofacceptance>2004-01-01</dateofacceptance>
-							<publisher>Den Haag: Boom</publisher>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">od_______323::92a8ab394fcf603e24b3ecc6e27fe2d1</to>
-							<similarity>0.45559114</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Asielbeleid speelt
-								continu op nationaal en Europees niveau</title>
-							<dateofacceptance>2011-01-01</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">od_______323::d8d6bbc2b32b757ccd9114366428482a</to>
-							<similarity>0.39992455</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Gezondheidszorg voor
-								illegale vreemdelingen in Nederland</title>
-							<dateofacceptance>2010-01-01</dateofacceptance>
-							<publisher>Bohn Stafleu van Loghum</publisher>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">phd_tilb__nl::1fa27b4261b757df6e26ba66159cd656</to>
-							<similarity>0.39992455</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Herziening Wet arbeid
-								vreemdelingen : een beschouwing vanuit Europees perspectief
-							</title>
-							<dateofacceptance>2013-01-01</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">phd_tilb__nl::a72bc9398100970fa0dd204a81cb454c</to>
-							<similarity>0.39992455</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">De
-								bijstandsrechtelijke positie van vreemdelingen in Nederland : de
-								relatie met internationale normen</title>
-							<dateofacceptance>1991-01-01</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">phd_tilb__nl::3185d852d8be25e762803eaac8abc1e0</to>
-							<similarity>0.35941428</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Wet arbeid
-								vreemdelingen : hoe werkt de Wet arbeid vreemdelingen in zowel
-								de theorie als in de praktijk en welke veranderingen zijn er
-								gewenst c.q. noodzakelijk?</title>
-							<dateofacceptance>2011-01-01</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">phd_tilb__nl::1b651d4468ab4f92eb283360ef76ea11</to>
-							<similarity>0.39992455</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Ruimte voor bestuur,
-								ruimte voor vreemdelingen</title>
-							<dateofacceptance>1995-01-01</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">phd_tilb__nl::82122004eb1325c61ba8642403a0baae</to>
-							<similarity>0.39992455</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">De au pair in relatie
-								tot de Wet arbeid vreemdelingen</title>
-							<dateofacceptance>2009-01-01</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">phd_tilb__nl::bd9c27d3091b0e3bf3fef53a5808fe8e</to>
-							<similarity>0.39992455</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Herziening Wet arbeid
-								vreemdelingen: theorie versus praktijk</title>
-							<dateofacceptance>2010-01-01</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">phd_tilb__nl::94c4b268d9451b23d49e1c4e8dcf7b8f</to>
-							<similarity>0.43279523</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">De positie van vrouwen
-								in het Nederlandse vreemdelingenbeleid</title>
-							<dateofacceptance>2004-01-01</dateofacceptance>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-						<rel inferred="true" trust="0.9"
-							inferenceprovenance="iis::document_similarities_standard"
-							provenanceaction="iis">
-							<to class="hasAmongTopNSimilarDocuments" scheme="dnet:result_result_relations"
-								type="result">od_______369::2bc5ccb12cc12f2db094124f0fa8b04d</to>
-							<similarity>0.45595452</similarity>
-							<type>STANDARD</type>
-							<title classid="main title" classname="main title"
-								schemeid="dnet:dataCite_title" schemename="dnet:dataCite_title">Regelrust voor
-								vreemdelingen. Een advies over vermindering van regeldruk in het
-								reguliere vreemdelingenbeleid.</title>
-							<dateofacceptance>2010-01-01</dateofacceptance>
-							<publisher>Den Haag</publisher>
-							<resulttype classid="publication" classname="publication"
-								schemeid="dnet:result_typologies" schemename="dnet:result_typologies" />
-						</rel>
-					</rels>
-					<children>
-						<instance id="driver______::7d8b320d781e5eced039bb46e6ec3c56">
-							<licence classid="OPEN" classname="Open Access" schemeid="dnet:access_modes"
-								schemename="dnet:access_modes" />
-							<instancetype classid="0017" classname="Report"
-								schemeid="dnet:publication_resource" schemename="dnet:publication_resource" />
-							<hostedby name="WODC Repository Ministerie van Veiligheid en Justitie"
-								id="driver______::7d8b320d781e5eced039bb46e6ec3c56" />
-							<webresource>
-								<url>http://resolver.tudelft.nl/uuid:017d6abc-dc6c-4560-b3bc-16bab777978e
-								</url>
-							</webresource>
-
-						</instance>
-
-					</children>
-				</oaf:result>
-				<counters>
-					<counter_authorship value="1" />
-					<counter_similarity value="20" />
-				</counters>
-			</oaf:entity>
-		</metadata>
-	</result>
-</record>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/dedup/TitleOrderingTest.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/dedup/TitleOrderingTest.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/dedup/TitleOrderingTest.java	(nonexistent)
@@ -1,91 +0,0 @@
-package eu.dnetlib.data.mapreduce.dedup;
-
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.PriorityQueue;
-import java.util.Queue;
-import java.util.UUID;
-
-import org.apache.commons.io.IOUtils;
-import org.junit.Before;
-import org.junit.Test;
-import org.springframework.core.io.ClassPathResource;
-
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-
-import eu.dnetlib.pace.clustering.NGramUtils;
-import eu.dnetlib.pace.config.Type;
-import eu.dnetlib.pace.model.FieldListImpl;
-import eu.dnetlib.pace.model.FieldValueImpl;
-import eu.dnetlib.pace.model.MapDocument;
-import eu.dnetlib.pace.model.MapDocumentComparator;
-
-public class TitleOrderingTest {
-
-	private List<MapDocument> results = Lists.newArrayList();
-
-	@Before
-	public void setUp() throws Exception {
-
-		final List<String> lines = IOUtils.readLines(new ClassPathResource("eu/dnetlib/data/mapreduce/dedup/titles.txt").getInputStream());
-		for (String title : lines) {
-			Map<String, FieldListImpl> fieldMap = Maps.newHashMap();
-			FieldListImpl list = new FieldListImpl();
-			list.add(new FieldValueImpl(Type.String, "title", title));
-			fieldMap.put("title", list);
-			results.add(new MapDocument("id-" + UUID.randomUUID(), fieldMap));
-		}
-	}
-
-	@Test
-	public void test() {
-
-		final Queue<MapDocument> queue = new PriorityQueue<MapDocument>(100, new MapDocumentComparator("title"));
-
-		queue.addAll(results);
-
-		Queue<MapDocument> queue2 = simplifyQueue(queue);
-
-		while (!queue2.isEmpty()) {
-			MapDocument doc = queue2.remove();
-			System.out.println(doc.values("title").stringValue());
-		}
-	}
-
-	private Queue<MapDocument> simplifyQueue(final Queue<MapDocument> queue) {
-		final Queue<MapDocument> q = new LinkedList<MapDocument>();
-
-		String fieldRef = "";
-		List<MapDocument> tempResults = Lists.newArrayList();
-
-		while (!queue.isEmpty()) {
-			MapDocument result = queue.remove();
-
-			if (!result.values("title").isEmpty()) {
-				String field = NGramUtils.cleanupForOrdering(result.values("title").stringValue());
-				if (field.equals(fieldRef)) {
-					tempResults.add(result);
-				} else {
-					if (tempResults.size() < 5) {
-						q.addAll(tempResults);
-					} else {
-						System.out.println("Skipped field: " + fieldRef + " - size: " + tempResults.size());
-					}
-					tempResults.clear();
-					tempResults.add(result);
-					fieldRef = field;
-				}
-			}
-		}
-		if (tempResults.size() < 5) {
-			q.addAll(tempResults);
-		} else {
-			System.out.println("Skipped field: " + fieldRef + " - size: " + tempResults.size());
-		}
-
-		return q;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/dedup/OafMergeTest.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/dedup/OafMergeTest.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/dedup/OafMergeTest.java	(nonexistent)
@@ -1,136 +0,0 @@
-package eu.dnetlib.data.mapreduce.dedup;
-
-import java.util.List;
-import java.util.UUID;
-
-import org.junit.Before;
-import org.junit.Test;
-
-import com.google.common.base.Predicate;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-
-import eu.dnetlib.data.mapreduce.util.OafTest;
-import eu.dnetlib.data.proto.FieldTypeProtos.StructuredProperty;
-import eu.dnetlib.data.proto.KindProtos.Kind;
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.proto.OafProtos.OafEntity;
-import eu.dnetlib.data.proto.OafProtos.OafEntity.Builder;
-import eu.dnetlib.data.proto.ResultProtos.Result;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.data.transform.OafEntityMerger;
-
-public class OafMergeTest {
-
-	private List<Oaf> oafList;
-
-	private OafEntityMerger merger;
-
-	@Before
-	public void setUp() throws Exception {
-
-		merger = new OafEntityMerger();
-		oafList = Lists.newArrayList();
-		oafList.add(getOaf("0.1").setEntity(
-				getEntity("id_1", null, "pid_1").setResult(
-						Result.newBuilder().setMetadata(
-								Result.Metadata.newBuilder().setDateofacceptance(OafTest.sf("2012-01-01"))
-										.addTitle(OafTest.getStructuredproperty("vvvv Title", "main title", "dnet:dataCite_title"))
-										.setResulttype(OafTest.getQualifier("publication", "dnet:result_typologies"))))).build());
-		oafList.add(getOaf("0.1").setEntity(
-				getEntity("id_2", "originalId_2", "pid_2").setResult(
-						Result.newBuilder().setMetadata(
-								Result.Metadata.newBuilder().setDateofacceptance(OafTest.sf(""))
-										.addTitle(OafTest.getStructuredproperty("aaaa Title", "main title", "dnet:dataCite_title"))
-										.setResulttype(OafTest.getQualifier("publication", "dnet:result_typologies"))))).build());
-		oafList.add(getOaf("0.2").setEntity(
-				getEntity("id_3", "originalId_2", "pid_2").setResult(
-						Result.newBuilder().setMetadata(
-								Result.Metadata.newBuilder().addTitle(OafTest.getStructuredproperty("cccc Title", "sub title", "dnet:dataCite_title"))
-										.setResulttype(OafTest.getQualifier("publication", "dnet:result_typologies"))))).build());
-
-		oafList.add(getOaf("0.3").setEntity(
-				getEntity("id_$", null, "pid_3").setResult(
-						Result.newBuilder().setMetadata(
-								Result.Metadata.newBuilder().setPublisher(OafTest.sf("AMER CHEMICAL SOCXXXXXXXXXXXXXXXXX"))
-										.setResulttype(OafTest.getQualifier("publication", "dnet:result_typologies"))))).build());
-		oafList.add(getOaf("0.5").setEntity(
-				getEntity("id_5", null, null).setResult(
-						Result.newBuilder().setMetadata(
-								Result.Metadata.newBuilder().addTitle(OafTest.getStructuredproperty("hhhh title", "main title", "dnet:dataCite_title"))
-										.setPublisher(OafTest.sf("AMER CHEMICAL SOC X"))
-										.setResulttype(OafTest.getQualifier("publication", "dnet:result_typologies")).setStoragedate(OafTest.sf("2012-11-18"))
-										.setLanguage(OafTest.getQualifier("eng", "dnet:languages")).addDescription(OafTest.sf("original description")))))
-				.build());
-		oafList.add(getOaf("0.6").setEntity(
-				getEntity("id_6", null, "pid_6").setResult(
-						Result.newBuilder().setMetadata(
-								Result.Metadata.newBuilder().setResulttype(OafTest.getQualifier("publication", "dnet:result_typologies"))
-										.addDescription(OafTest.sf("new description"))))).build());
-	}
-
-	@Test
-	public void test_merge() {
-
-		final Oaf.Builder builder = Oaf.newBuilder();
-
-		for (final Oaf oaf : oafList) {
-			builder.mergeFrom(oaf);
-		}
-
-		final Result.Metadata.Builder metadata = builder.getEntityBuilder().getResultBuilder().getMetadataBuilder();
-		final Iterable<StructuredProperty> filter = Iterables.filter(metadata.getTitleList(), new Predicate<StructuredProperty>() {
-
-			@Override
-			public boolean apply(final StructuredProperty sp) {
-				return (sp.getQualifier() != null) && sp.getQualifier().getClassname().equals("main title");
-			}
-		});
-
-		final StructuredProperty last = Iterables.getLast(filter);
-
-		metadata.clearTitle().addAllTitle(Lists.newArrayList(last));
-
-		System.out.println(builder.build().toString());
-	}
-
-	@Test
-	public void test_merger() {
-
-		final Oaf merge = merger.mergeEntities(null, "id", oafList).build();
-
-		System.out.println(merge.toString());
-	}
-
-	// @Test
-	// public void test_sort() {
-	// Queue<Result> q = new PriorityQueue<Result>(3, DedupReducer.cmp);
-	// for (Oaf oaf : oafList) {
-	// q.add(oaf.getEntity().getResult());
-	// }
-	//
-	// while (!q.isEmpty()) {
-	// Result r = q.remove();
-	// List<StructuredProperty> titles = r.getMetadata().getTitleList();
-	// if (!titles.isEmpty()) {
-	// System.out.println(titles.get(0).getValue());
-	// }
-	// }
-	// }
-
-	private Oaf.Builder getOaf(final String trust) {
-		return Oaf.newBuilder().setKind(Kind.entity).setDataInfo(OafTest.getDataInfo(trust)).setTimestamp(System.currentTimeMillis());
-	}
-
-	private OafEntity.Builder getEntity(final String id, final String originalId, final String pid) {
-		final Builder entity =
-				OafEntity.newBuilder().setType(Type.result).setId(id).addOriginalId(originalId != null ? originalId : UUID.randomUUID().toString());
-
-		if (pid != null) {
-			entity.addPid(OafTest.getStructuredproperty(pid, "class", "scheme"));
-		}
-
-		return entity;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/Context.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/Context.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/Context.java	(nonexistent)
@@ -1,712 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index.config;
-
-public class Context {
-
-	public final static String xml = "<ContextDSResources>" + "<entry id=\"egi\" label=\"EGI\" name=\"context\" type=\"community\"/>"
-			+ "<entry id=\"egi::classification\" label=\"Scientific Disciplines\" name=\"category\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc\" label=\"Natural Sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::math\" label=\"Mathematics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::math::applied\" label=\"Applied Mathematics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::math::pure\" label=\"Pure Mathematics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::math::stats\" label=\"Statistics and Probability\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::math::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc\" label=\"Computer sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::algo\" label=\"Algorithms\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::artii\" label=\"Artificial Intelligence\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::carc\" label=\"Computer architecture\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::ccomm\" label=\"Computer communications\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::cgraph\" label=\"Computer graphics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::csecr\" label=\"Computer security and reliability\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::dstr\" label=\"Data structures\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::disc\" label=\"Distributed computing\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::hci\" label=\"Human-computer interaction\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::opes\" label=\"Operating systems\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::parc\" label=\"Parallel computing\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::plan\" label=\"Programming languages\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::qcom\" label=\"Quantum computing\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::softe\" label=\"software engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::theo\" label=\"Theory of computation\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::cosc::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::infos\" label=\"Information sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::infos::dman\" label=\"Data management\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::infos::dmin\" label=\"Data mining\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::infos::iret\" label=\"Information retrieval\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::infos::iman\" label=\"Information management\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::infos::kman\" label=\"Knowledge management\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::infos::mult\" label=\"Multimedia, hypermedia\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::infos::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths\" label=\"Earth sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::atms\" label=\"Atmospheric science\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::clim\" label=\"Climate research\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::geoc\" label=\"Geochemistry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::geol\" label=\"Geology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::geop\" label=\"Geophysics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::hydr\" label=\"Hydrology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::mine\" label=\"Mineralogy\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::ocea\" label=\"Oceanography\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::pala\" label=\"palaeontology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::pgeo\" label=\"Physical geography\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::seis\" label=\"Seismology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::volc\" label=\"Volcanology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::earths::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc\" label=\"Biological sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::aerob\" label=\"Aerobiology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::bact\" label=\"Bacteriology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::behav\" label=\"Behavioural biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::bimol\" label=\"Biochemistry and molecular biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::bcons\" label=\"Biodiversity conservation\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::binfo\" label=\"Bioinformatics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::brhy\" label=\"Biological rhythm\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::biol\" label=\"Biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::bioph\" label=\"Biophysics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::bota\" label=\"Botany\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::cellb\" label=\"Cell biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::cobio\" label=\"Computational biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::cryo\" label=\"Cryobiology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::devb\" label=\"Developmental biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::ecol\" label=\"Ecology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::evolb\" label=\"Evolutionary biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::gehe\" label=\"Genetics and heredity\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::marin\" label=\"Marine and Freshwater biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::mbio\" label=\"Mathematical biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::mibio\" label=\"Microbiology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::myco\" label=\"Mycology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::plant\" label=\"Plant science\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::repr\" label=\"Reproductive biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::strub\" label=\"Structural biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::taxo\" label=\"Taxonomy\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::theob\" label=\"Theoretical biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::therm\" label=\"Thermal biology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::viro\" label=\"Virology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::zool\" label=\"Zoology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::biosc::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc\" label=\"Physical sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::acce\" label=\"Acceleration physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::acou\" label=\"Acoustics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::aero\" label=\"Aerosol physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::astrb\" label=\"Astrobiology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::astrn\" label=\"Astronomy\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::astrp\" label=\"Astroparticle physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::astrph\" label=\"Astrophysics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::atmc\" label=\"Atomic\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::chph\" label=\"Chemical physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::comph\" label=\"Computational physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::conds\" label=\"Condensed matter physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::cryog\" label=\"Cryogenics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::fluid\" label=\"Fluid mechanics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::fusi\" label=\"Fusion\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::hene\" label=\"High energy physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::mathp\" label=\"Mathematical physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::medip\" label=\"Medical physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::molep\" label=\"Molecular physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::nuclp\" label=\"Nuclear physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::optc\" label=\"Optics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::partp\" label=\"Particle physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::phys\" label=\"Physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::plans\" label=\"Planetary science\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::plsm\" label=\"Plasma physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::spac\" label=\"Space physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::quant\" label=\"Quantum physics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::physc::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::chesc\" label=\"Chemical sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::chesc::analc\" label=\"Analytical chemistry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::chesc::chem\" label=\"Chemistry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::chesc::collc\" label=\"Colloid chemistry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::chesc::compc\" label=\"Computational chemistry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::chesc::elecc\" label=\"Electrochemistry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::chesc::innuc\" label=\"Inorganic and nuclear chemistry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::chesc::mathc\" label=\"Mathematical chemistry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::chesc::orgc\" label=\"Organic chemistry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::chesc::phyc\" label=\"Physical chemistry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::chesc::polc\" label=\"Polymer chemistry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::natsc::chesc::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt\" label=\"Engineering and Technology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::cieng\" label=\"Civil engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::civeng::arche\" label=\"Architecture engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::civeng::cieng\" label=\"Civil engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::civeng::cipro\" label=\"Civil protection\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::civeng::costr\" label=\"Construction/Structural engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::civeng::treng\" label=\"Transport engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::civeng::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::eeie\" label=\"Electrical, electronic and information engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::eeie::comes\" label=\"Communication engineering and systems\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::eeie::comha\" label=\"Computer hardware and architecture\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::eeie::elele\" label=\"Electrical and electronic engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::eeie::robaut\" label=\"Robotics, Automation and Control Systems\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::eeie::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::meng\" label=\"Mechanical engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::meng::apmec\" label=\"Applied mechanics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::meng::audm\" label=\"Audio engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::meng::nucl\" label=\"Nuclear related engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::meng::relia\" label=\"Reliability analysis\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::meng::therd\" label=\"Thermodynamics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::meng::other\" label=\"other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::aeros\" label=\"Aerospace engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::aeros::aeren\" label=\"Aeronautical engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::aeros::astren\" label=\"Astronautical engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::aeros::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::cheme\" label=\"Chemical engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::cheme::cheme\" label=\"Chemical engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::cheme::chempr\" label=\"Chemical process engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::cheme::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::maten\" label=\"Materials engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::maten::ceram\" label=\"Ceramics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::maten::coatf\" label=\"Coating and films\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::maten::compos\" label=\"Composites\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::maten::paperw\" label=\"Paper and wood\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::maten::text\" label=\"Textiles\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::maten::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::bioeng\" label=\"Bioengineering and Biomedical engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::bioeng::bioeng\" label=\"Bioengineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::bioeng::biomed\" label=\"Biomedical engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::bioeng::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::enveng\" label=\"Environmental engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::enveng::enfue\" label=\"Energy and fuels\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::enveng::geoen\" label=\"Geological engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::enveng::geote\" label=\"Geotechnics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::enveng::oceae\" label=\"Ocean engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::enveng::minep\" label=\"Mining and mineral processing\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::enveng::petren\" label=\"Petroleum engineering\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::enveng::remse\" label=\"Remote sensing\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::enveng::seav\" label=\"Sea vessels\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::enveng::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::envbio\" label=\"Environmental biotechnology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::envbio::biore\" label=\"Bioremediation\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::envbio::diabi\" label=\"Diagnostic biotechnologies\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::envbio::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::indbio\" label=\"Industrial  biotechnology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::indbio::biden\" label=\"Bio-derived novel materials\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::indbio::bicat\" label=\"Biocatalysis\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::indbio::bibul\" label=\"Bioderived bulk and fine chemicals\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::indbio::bifue\" label=\"Biofuels\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::indbio::bimat\" label=\"Biomaterials\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::indbio::bipro\" label=\"Bioprocessing technologies\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::indbio::biprd\" label=\"Bioproducts\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::indbio::ferm\" label=\"Fermentation\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::indbio::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::nante\" label=\"Nano-technology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::nante::nanma\" label=\"Nano-materials\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::nante::nanpr\" label=\"Nano-processes\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::engt::nante::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc\" label=\"Medical and Health Sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::basicm\" label=\"Basic medicine\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::basicm::anatm\" label=\"Anatomy and morphology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::basicm::hugen\" label=\"Human genetics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::basicm::immu\" label=\"Immunology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::basicm::medch\" label=\"Medicinal chemistry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::basicm::neusc\" label=\"Neuroscience\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::basicm::path\" label=\"Pathology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::basicm::pharm\" label=\"Pharmacology and pharmacy\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::basicm::physi\" label=\"Physiology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::basicm::toxi\" label=\"Toxicology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::basicm::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm\" label=\"Clinical medicine\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::alle\" label=\"Allergy\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::anae\" label=\"Anaesthesiology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::andro\" label=\"Andrology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::cardi\" label=\"Cardiac and Cardiovascular systems\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::emerg\" label=\"Critical care/Emergency medicine\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::dent\" label=\"Dentistry, oral surgery/medicine\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::derm\" label=\"Dermatology and venereal diseases\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::gastr\" label=\"Gastroenterology and hepatology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::gener\" label=\"General and internal medicine\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::geria\" label=\"Geriatrics and gerontology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::hema\" label=\"Hematology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::compl\" label=\"Integrative and Complementary medicine\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::medim\" label=\"Medical imaging\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::nucim\" label=\"Nuclear medicine\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::obstg\" label=\"Obstetrics and gynaecology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::oncol\" label=\"Oncology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::ophth\" label=\"Ophthalmology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::optom\" label=\"Optometry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::ortho\" label=\"Orthopaedics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::otori\" label=\"Otorhinolaryngology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::paedi\" label=\"Paediatrics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::periph\" label=\"Peripheral vascular disease\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::psych\" label=\"Psychiatry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::radio\" label=\"Radiology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::respi\" label=\"Respiratory systems\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::rheum\" label=\"Rheumatology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::surge\" label=\"Surgery\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::transp\" label=\"Transplantation\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::urol\" label=\"Urology and nephrology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::clinm::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc\" label=\"Health science\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::epide\" label=\"Epidemiology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::caresc\" label=\"Health care science and services\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::policy\" label=\"Health policy and services\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::infec\" label=\"Infectious diseases\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::ethic\" label=\"Medical ethics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::nurse\" label=\"Nursing\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::nutri\" label=\"Nutrition and Dietetics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::occup\" label=\"Occupational health\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::paras\" label=\"Parasitology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::publi\" label=\"Public and environmental health\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::sobio\" label=\"Social biomedical science\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::sport\" label=\"Sport and fitness science\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::subst\" label=\"Substance abuse\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::trop\" label=\"Tropical medicine\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::healsc::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::medbio\" label=\"Medical biotechnology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::medbio::bdev\" label=\"Biomedical devices\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::medbio::hbio\" label=\"Health-related biotechnology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::medbio::phbio\" label=\"Pharmaceutical biotechnology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::medbio::bioeth\" label=\"Biotechnology and medical ethics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::medbio::mdia\" label=\"Molecular diagnostics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::medbio::bmani\" label=\"Biophysical manipulation\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::medsc::medbio::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc\" label=\"Agricultural Sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrfor\" label=\"Agriculture, forestry, and fisheries\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrfor::agri\" label=\"Agriculture\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrfor::agrpl\" label=\"Agronomy, plant breeding, plant protection\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrfor::fish\" label=\"Fishery\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrfor::fore\" label=\"Forestry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrfor::horti\" label=\"Horticulture and viticulture\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrfor::soil\" label=\"Soil science\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrfor::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::anisc\" label=\"Animal and dairy sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::anisc::anisc\" label=\"Animal science\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::anisc::dair\" label=\"Dairy science\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::anisc::husb\" label=\"Husbandry\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::anisc::pets\" label=\"Pets\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::anisc::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::vetsc\" label=\"Veterinary sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::vetsc::vanae\" label=\"Veterinary anaesthesiology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::vetsc::vmed\" label=\"Veterinary medicine\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::vetsc::vophth\" label=\"Veterinary ophtahlmology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::vetsc::vpath\" label=\"Veterinary pathobiology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::vetsc::vrad\" label=\"Veterinary radiology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::vetsc::vrepr\" label=\"Veterinary reproduction\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::vetsc::vsurg\" label=\"Veterinary surgery\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::vetsc::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrbio\" label=\"Agricultural biotechnology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrbio::bmass\" label=\"Biomass feedstock production tech.\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrbio::bphar\" label=\"Biopharming\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrbio::diagn\" label=\"Diagnostics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrbio::fbiot\" label=\"Food biotechnology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrbio::gmtec\" label=\"GM technology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrbio::lclon\" label=\"Livestock cloning\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrbio::masel\" label=\"Marker assisted selection\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::agrisc::agrbio::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc\" label=\"Social Sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych\" label=\"Psychology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych::biops\" label=\"Biological Psychology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych::clips\" label=\"Clinical Psychology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych::cogps\" label=\"Cognitive Psychology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych::comps\" label=\"Comparative Psychology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych::devps\" label=\"Developmental Psychology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych::edups\" label=\"Educational and School Psychology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych::evops\" label=\"Evolutionary Psychology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych::indps\" label=\"Industrial-organisational Psychology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych::persps\" label=\"Personality Psychology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych::posps\" label=\"Positive Psychology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych::socps\" label=\"Social Psychology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::psych::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::econ\" label=\"Economics, finance and business\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::econ::busman\" label=\"Business and management\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::econ::econom\" label=\"Economics and Econometrics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::econ::finan\" label=\"Finance\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::econ::indus\" label=\"Industrial relations\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::econ::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::edusc\" label=\"Educational sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::edusc::genedu\" label=\"General Education\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::edusc::spedu\" label=\"Special Education\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::edusc::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::socio\" label=\"Sociology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::socio::anthro\" label=\"Anthropology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::socio::demog\" label=\"Demography\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::socio::ethno\" label=\"Ethnology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::socio::famil\" label=\"Family studies\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::socio::socst\" label=\"Social studies\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::socio::sowork\" label=\"Social work\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::socio::sociol\" label=\"Sociology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::socio::women\" label=\"Women's and gender studies\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::socio::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::law\" label=\"Law\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::law::canol\" label=\"Canon Law\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::law::civill\" label=\"Civil Law\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::law::comparl\" label=\"Comparative Law\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::law::compel\" label=\"Competition Law\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::law::constl\" label=\"Constitutional Law\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::law::crimil\" label=\"Criminal Law\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::law::islaml\" label=\"Islamic Law\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::law::jewil\" label=\"Jewish Law\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::law::juris\" label=\"Jurisprudence\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::law::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::polsc\" label=\"Political Sciences\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::polsc::compp\" label=\"Comparative Politics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::polsc::empir\" label=\"Empirical pata analysis\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::polsc::intre\" label=\"International relations\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::polsc::organ\" label=\"Organisational theory\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::polsc::polec\" label=\"Political economy\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::polsc::polph\" label=\"Political philosophy\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::polsc::pubad\" label=\"Public administration\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::polsc::thsta\" label=\"Theories of the state\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::polsc::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::soecge\" label=\"Social and economic geography\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::soecge::cuecge\" label=\"Cultural and economic geography\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::soecge::transp\" label=\"Transport planning\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::soecge::urbst\" label=\"Urban studies\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::soecge::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::media\" label=\"Media and communications\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::media::cuecge\" label=\"Information science - social\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::media::journ\" label=\"Journalism\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::media::librsc\" label=\"Library science\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::media::mediasoc\" label=\"Media and socio-cultural communication\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::sosc::media::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human\" label=\"Humanities\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::hist\" label=\"History and Archaeology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::hist::archae\" label=\"Archaeology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::hist::hist\" label=\"History\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::hist::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::lang\" label=\"Languages and literature\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::lang::genlang\" label=\"General language studies\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::lang::genlit\" label=\"General literature studies\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::lang::ling\" label=\"Linguistics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::lang::liter\" label=\"Literary theory\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::lang::splang\" label=\"Specific languages\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::lang::split\" label=\"Specific literatures\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::lang::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::philos\" label=\"Philosophy, ethics and religion\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::philos::ethic\" label=\"Ethics\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::philos::philsc\" label=\"Philosophy of science/technology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::philos::philo\" label=\"Philosophy\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::philos::relig\" label=\"Religious studies\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::philos::theol\" label=\"Theology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::philos::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::arts\" label=\"Arts\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::arts::archi\" label=\"Architectural design\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::arts::folk\" label=\"Folklore Studies\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::arts::media\" label=\"Media Studies\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::arts::music\" label=\"Musicology\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::arts::perf\" label=\"Performing arts studies\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::human::arts::Other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::supp\" label=\"Support Activities\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::supp::digarc\" label=\"Digital Archives\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::supp::infra\" label=\"Infrastructure Development\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::supp::train\" label=\"Training/Demonstrations\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::classification::supp::other\" label=\"Other\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects\" label=\"EGI Projects\" name=\"category\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::biomedbridges\" label=\"BioMedBridges\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::biovel\" label=\"BioVel\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::chain\" label=\"CHAIN\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::dchrp\" label=\"DCH-RP\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::decide\" label=\"DECIDE\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::drihms\" label=\"DRIHMS\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::efiscal\" label=\"e-Fiscal\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::esciencetalk\" label=\"e-ScienceTalk\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::ige\" label=\"IGE\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::enventory\" label=\"e.NVENTORY\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::egdi\" label=\"EDGI\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::egiinspire\" label=\"EGI-InSPIRE\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::emi\" label=\"EMI\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::envri\" label=\"ENVRI\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::erflow\" label=\"ER-Flow\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::mantychore\" label=\"MANTYCHORE\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::mapper\" label=\"MAPPER\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::scalalife\" label=\"SCALALIFE\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::scibus\" label=\"SCI-BUS\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::shiwa\" label=\"SHIWA\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::siena\" label=\"SIENA\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::stratuslab\" label=\"STRATUSLAB\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::uvasce\" label=\"UVACSE\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::verce\" label=\"VERCE\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::wenmr\" label=\"WENMR\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::helixn\" label=\"Helix Nebula\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::projects::fedsm\" label=\"FedSM\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country\" label=\"Countries\" name=\"category\" type=\"\"/>"
-			+ "<entry id=\"egi::country::ba\" label=\"Bosnia and Herzegovina\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::be\" label=\"Belgium\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::bg\" label=\"Bulgaria\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::ch\" label=\"Switzerland\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::cy\" label=\"Cyprus\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::cz\" label=\"Czech Republic\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::de\" label=\"Germany\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::dk\" label=\"Denmark\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::ee\" label=\"Estonia\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::es\" label=\"Spain\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::fi\" label=\"Finland\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::fr\" label=\"France\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::gr\" label=\"Greece\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::hr\" label=\"Croatia\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::hu\" label=\"Hungary\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::ie\" label=\"Ireland\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::il\" label=\"Israel\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::it\" label=\"Italy\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::lt\" label=\"Lithuania\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::lu\" label=\"Luxembourg\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::lv\" label=\"Latvia\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::me\" label=\"Montenegro\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::mk\" label=\"Macedonia\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::nl\" label=\"the Netherlands\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::nr\" label=\"Norway\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::pl\" label=\"Poland\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::pt\" label=\"Portugal\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::ro\" label=\"Romania\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::rs\" label=\"Serbia\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::si\" label=\"Slovenia\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::sk\" label=\"Slovakia\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::se\" label=\"Sweden\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::tr\" label=\"Turkey\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::uk\" label=\"United Kingdom\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::am\" label=\"ARmenia\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::md\" label=\"Moldova\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::country::ru\" label=\"Russia\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation\" label=\"EGI Members\" name=\"category\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::etfbl\" label=\"ETFBL\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::belnet\" label=\"BELNET\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::iictbas\" label=\"IICT-BAS\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::swing\" label=\"SWiNG\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::cygrid\" label=\"CyGrid\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::cesnet\" label=\"CESNET\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::gaussallianz\" label=\"Gauss-Allianz\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::dcsc\" label=\"DCSC\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::eenet\" label=\"EENet\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::csic\" label=\"CSIC\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::csc\" label=\"CSC\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::cnrs\" label=\"CNRS\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::grnet\" label=\"GRNET\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::srce\" label=\"SRCE\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::niif\" label=\"NIIF\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::gridireland\" label=\"Grid-Ireland\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::iucc\" label=\"IUCC\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::infn\" label=\"INFN\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::vu\" label=\"VU\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::restena\" label=\"RESTENA\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::lumii\" label=\"LUMII\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::uom\" label=\"UoM\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::margi\" label=\"MARGI\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::ncf\" label=\"NCF\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::uninett\" label=\"UNINETT Sigma AS\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::cyfronet\" label=\"CYFRONET AGH\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::umic\" label=\"umic\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::ici\" label=\"ICI\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::ipb\" label=\"IPB\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::arnes\" label=\"ARNES\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::slovakgrid\" label=\"SlovakGrid\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::snic\" label=\"SNIC\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::ulakbim\" label=\"ULAKBIM\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::jisc\" label=\"JISC\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::iiapnas\" label=\"IIAP NAS RA\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::renam\" label=\"RENAM\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::earena\" label=\"e-ARENA\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::embl\" label=\"EMBL\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::organisation::cern\" label=\"CERN\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual\" label=\"EGI virtual organizations\" name=\"category\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2352\" label=\"vo.grand-est.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::41\" label=\"astron\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::26\" label=\"theophys\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::504\" label=\"vo.londongrid.ac.uk\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::78\" label=\"vo.grif.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1670\" label=\"eo-grid.ikd.kiev.ua\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2332\" label=\"ops.ndgf.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1430\" label=\"vo.msfg.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::40\" label=\"ilc\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2592\" label=\"gr-sim.grid.auth.gr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2172\" label=\"chem.vo.ibergrid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::311\" label=\"vo.sbg.in2p3.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10273\" label=\"comet.j-parc.jp\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10258\" label=\"vo.earthserver.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10249\" label=\"vo.sn2ns.in2p3.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::96\" label=\"trgridd\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::710\" label=\"vo.pic.es\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::13\" label=\"zeus\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::106\" label=\"supernemo.vo.eu-egee.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::6\" label=\"biomed\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::890\" label=\"prod.vo.eu-eela.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1030\" label=\"vo.ops.csic.es\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::250\" label=\"euindia\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::476\" label=\"vo.northgrid.ac.uk\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10283\" label=\"drihm.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2153\" label=\"ict.vo.ibergrid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10284\" label=\"hyperk.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2312\" label=\"vo.eu-decide.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1250\" label=\"superbvo.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::494\" label=\"gilda\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10252\" label=\"xenon.biggrid.nl\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::130\" label=\"balticgrid\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10265\" label=\"gerda.mpg.de\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::67\" label=\"belle\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2252\" label=\"vo.stratuslab.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10026\" label=\"camont\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10285\" label=\"peachnote.com\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::98\" label=\"ific\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1190\" label=\"pacs.infn.it\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2192\" label=\"oxgrid.ox.ac.uk\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::810\" label=\"glast.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2372\" label=\"demo.vo.edges-grid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2052\" label=\"vo.dorii.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1110\" label=\"vo.u-psud.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::479\" label=\"vo.cta.in2p3.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1630\" label=\"hgdemo\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::31\" label=\"ngs.ac.uk\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10260\" label=\"mpi-kickstart.egi.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1912\" label=\"ops.vo.ibergrid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::97\" label=\"trgride\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2472\" label=\"vo.indicate-project.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::411\" label=\"vo.lpnhe.in2p3.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::49\" label=\"planck\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::7\" label=\"egeode\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2492\" label=\"vo.sim-e-child.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2212\" label=\"uniandes.edu.co\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::910\" label=\"oper.vo.eu-eela.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10251\" label=\"vo.france-grilles.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::20\" label=\"ncf\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10268\" label=\"vo.astro.pic.es\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::570\" label=\"vo.hess-experiment.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1650\" label=\"sgdemo\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2152\" label=\"phys.vo.ibergrid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::22\" label=\"infngrid\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::69\" label=\"embrace\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10250\" label=\"bbmri.nl\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10083\" label=\"gridit\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10257\" label=\"vo.aginfra.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2432\" label=\"shiwa-workflow.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10267\" label=\"igi.italiangrid.it\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1450\" label=\"armgrid.grid.am\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::104\" label=\"gridcc\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1510\" label=\"vo.turbo.pic.es\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::37\" label=\"hone\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2176\" label=\"eng.vo.ibergrid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2174\" label=\"earth.vo.ibergrid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10287\" label=\"projects.nl\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::107\" label=\"auger\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::670\" label=\"vo.lpsc.in2p3.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::90\" label=\"eela\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::94\" label=\"trgridb\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::81\" label=\"proactive\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::990\" label=\"desktopgrid.vo.edges-grid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1610\" label=\"vo.delphi.cern.ch\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10262\" label=\"vo.gridcl.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::44\" label=\"ildg\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1710\" label=\"bg-edu.grid.acad.bg\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::36\" label=\"desy\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::3\" label=\"lhcb\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::5\" label=\"esr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::25\" label=\"inaf\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::970\" label=\"vo.grid.auth.gr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::89\" label=\"argo\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::58\" label=\"see\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10282\" label=\"kzvo.isragrid.org.il\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10271\" label=\"neurogrid.incf.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::150\" label=\"gaussian\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::62\" label=\"pheno\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::39\" label=\"hermes\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10256\" label=\"mice\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::77\" label=\"vo.lal.in2p3.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::15\" label=\"magic\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1952\" label=\"vo.neugrid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1933\" label=\"iber.vo.ibergrid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::473\" label=\"vo.scotgrid.ac.uk\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::60\" label=\"geant4\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::79\" label=\"vo.ipno.in2p3.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::76\" label=\"apesci\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::14\" label=\"compchem\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::68\" label=\"ops\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::461\" label=\"vo.agata.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10270\" label=\"na62.vo.gridpp.ac.uk\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::452\" label=\"seegrid\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2552\" label=\"snoplus.snolab.ca\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::497\" label=\"vo.irfu.cea.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::110\" label=\"gridpp\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10254\" label=\"ukmhd.ac.uk\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::830\" label=\"euasia.euasiagrid.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2392\" label=\"vo.mure.in2p3.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::46\" label=\"icecube\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::85\" label=\"ghep\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::19\" label=\"pvier\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10275\" label=\"neiss.org.uk\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::271\" label=\"eumed\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1730\" label=\"moldyngrid\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10259\" label=\"israelvo.isragrid.org.il\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::75\" label=\"twgrid\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::467\" label=\"enmr.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::950\" label=\"xfel.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2\" label=\"alice\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::64\" label=\"hungrid\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::105\" label=\"euchina\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1330\" label=\"meteo.see-grid-sci.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1270\" label=\"vlemed\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::482\" label=\"vo.gear.cern.ch\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2512\" label=\"vo.aleph.cern.ch\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1570\" label=\"vo.complex-systems.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1\" label=\"atlas\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::11\" label=\"virgo\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::47\" label=\"enea\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::88\" label=\"aegis\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::8\" label=\"cdf\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10276\" label=\"epic.vo.gridpp.ac.uk\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::610\" label=\"vo.southgrid.ac.uk\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2173\" label=\"life.vo.ibergrid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1050\" label=\"vo.general.csic.es\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1932\" label=\"tut.vo.ibergrid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10255\" label=\"vo.france-asia.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1150\" label=\"vo.up.pt\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1410\" label=\"lattice.itep.ru\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1891\" label=\"belle2.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::61\" label=\"auvergrid\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::71\" label=\"fusion\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1810\" label=\"vo.panda.gsi.de\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2292\" label=\"comput-er.it\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::503\" label=\"nordugrid.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::443\" label=\"gridmosi.ici.ro\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::4\" label=\"cms\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10263\" label=\"icarus-exp.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::630\" label=\"vo.renabi.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1770\" label=\"env.see-grid-sci.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::437\" label=\"vo.apc.univ-paris7.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::82\" label=\"calice\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2072\" label=\"vo.metacentrum.cz\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::391\" label=\"vo.llr.in2p3.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10269\" label=\"vo.ingv.it\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1790\" label=\"ams02.cern.ch\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::84\" label=\"dech\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::93\" label=\"trgrida\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2272\" label=\"envirogrids.vo.eu-egee.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::16\" label=\"dteam\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::92\" label=\"cesga\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::446\" label=\"imath.cesga.es\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::65\" label=\"uscms\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::108\" label=\"pamela\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::930\" label=\"vo.formation.idgrilles.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::790\" label=\"vo.rhone-alpes.idgrilles.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::730\" label=\"vo.paus.pic.es\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10274\" label=\"fedcloud.egi.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1972\" label=\"vo.lpta.in2p3.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::590\" label=\"d4science.research-infrastructures.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10278\" label=\"isravo.isragrid.org.il\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::491\" label=\"cppm\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::440\" label=\"astro.vo.eu-egee.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10288\" label=\"km3net.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10266\" label=\"ipv6.hepix.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1690\" label=\"atlas.ac.il\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1170\" label=\"vo.ipnl.in2p3.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2012\" label=\"bing.vo.ibergrid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::488\" label=\"vo.lapp.in2p3.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::66\" label=\"swetest\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1870\" label=\"vo.mcia.fr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1892\" label=\"cometa\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::91\" label=\"edteam\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10272\" label=\"verce.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::251\" label=\"lofar\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::458\" label=\"vo.plgrid.pl\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10261\" label=\"vo.landslides.mossaic.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10\" label=\"dzero\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1390\" label=\"lsgrid\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10286\" label=\"cernatschool.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::17\" label=\"ukqcd\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1310\" label=\"vo.helio-vo.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::870\" label=\"fkppl.kisti.re.kr\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2132\" label=\"vo.ifisc.csic.es\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10277\" label=\"vo.dch-rp.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10253\" label=\"dream.hipcat.net\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::102\" label=\"nw_ru\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2175\" label=\"social.vo.ibergrid.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10281\" label=\"lsst\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::57\" label=\"voce\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::12\" label=\"vo.sixt.cern.ch\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::2112\" label=\"seismo.see-grid-sci.eu\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1890\" label=\"vo.cs.br\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::1530\" label=\"t2k.org\" name=\"concept\" type=\"\"/>"
-			+ "<entry id=\"egi::virtual::10264\" label=\"net.egi.eu\" name=\"concept\" type=\"\"/>" + "</ContextDSResources>";
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/ContextMapperTest.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/ContextMapperTest.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/ContextMapperTest.java	(nonexistent)
@@ -1,49 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index.config;
-
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertTrue;
-
-import java.util.Map.Entry;
-
-import org.dom4j.DocumentException;
-import org.junit.Before;
-import org.junit.Test;
-
-public class ContextMapperTest {
-
-	private ContextMapper contextMapper;
-
-	@Before
-	public void setUp() throws Exception {
-		contextMapper = ContextMapper.fromXml(Context.xml);
-		assertNotNull(contextMapper);
-	}
-
-	@Test
-	public void test() throws DocumentException {
-		assertTrue(contextMapper.size() > 0);
-
-		for (Entry<String, ContextDef> entry : contextMapper.entrySet()) {
-			ContextDef def = entry.getValue();
-
-			try {
-				assertNotNull(def.getId());
-				assertNotNull(def.getLabel());
-				assertNotNull(def.getName());
-
-				assertFalse(def.getId().isEmpty());
-				assertFalse(def.getLabel().isEmpty());
-				assertFalse(def.getName().isEmpty());
-
-				if (def.getName().equals("context")) {
-					assertNotNull(def.getType());
-					assertFalse(def.getType().isEmpty());
-				}
-			} catch (Throwable e) {
-				System.out.println("aaa");
-			}
-		}
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/IndexConfigTest.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/IndexConfigTest.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/IndexConfigTest.java	(nonexistent)
@@ -1,94 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index.config;
-
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertTrue;
-
-import org.junit.Test;
-
-import eu.dnetlib.data.mapreduce.util.RelDescriptor;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-
-public class IndexConfigTest {
-
-	public static final String config =
-			"index.conf { "
-					+ "result { dups = true, links = ["
-					+ "{ relType = personResult_authorship_hasAuthor, targetEntity = person, expandAs = rel, symmetric = true, fields = [title,dateofacceptance,publisher,resulttype] }, "
-					+ "{ relType = resultResult_dedup_isMergedIn, targetEntity = result, expandAs = child, symmetric = true, fields = [title,dateofacceptance,publisher,resulttype,pid] }, "
-					+ "{ relType = resultResult_dedup_merges, targetEntity = result, expandAs = child, symmetric = true, fields = [title,dateofacceptance,publisher,resulttype,pid] }, "
-					+ "{ relType = resultResult_publicationDataset_isRelatedTo, targetEntity = result, expandAs = rel, symmetric = true, fields = [title,dateofacceptance,publisher,resulttype,pid] },"
-					+ "{ relType = resultResult_similarity_isAmongTopNSimilarDocuments, targetEntity = result, expandAs = rel, symmetric = true, fields = [title,dateofacceptance,publisher,resulttype,similarity,type,pid] },"
-					+ "{ relType = resultResult_similarity_hasAmongTopNSimilarDocuments, targetEntity = result, expandAs = rel, symmetric = true, fields = [title,dateofacceptance,publisher,resulttype,similarity,type,pid] }"
-					+ "]}, "
-					+ "person { dups = false, links = ["
-					+ "{ relType = personResult_authorship_isAuthorOf, targetEntity = result, expandAs = rel, symmetric = true, fields = [fullname,ranking], max = 2 }, "
-					+ "{ relType = projectPerson_contactPerson_isContact, targetEntity = project, expandAs = rel, symmetric = true, fields = [fullname,email,fax,phone] } "
-					+ "]}, "
-					+ "datasource { dups = false, links = ["
-					+ "{ relType = datasourceOrganization_provision_provides, targetEntity = organization, expandAs = rel, symmetric = true, fields = [officialname,websiteurl,datasourcetype,aggregatortype] } "
-					+ "]}, "
-					+ "organization { dups = false, links = ["
-					+ "{ relType = projectOrganization_participation_isParticipant, targetEntity = project, expandAs = rel, symmetric = true, fields = [legalname,legalshortname,websiteurl,country] }, "
-					+ "{ relType = datasourceOrganization_provision_isProvidedBy, targetEntity = datasource, expandAs = rel, symmetric = true, fields = [legalname,legalshortname,websiteurl,country] }, "
-					+ "{ relType = organizationOrganization_dedup_merges, targetEntity = organization, expandAs = child, symmetric = true, fields = [legalname,legalshortname,websiteurl,country] }, "
-					+ "{ relType = organizationOrganization_dedup_isMergedIn, targetEntity = organization, expandAs = child, symmetric = true, fields = [legalname,legalshortname,websiteurl,country] }"
-					+ "]}, "
-					+ "project { dups = false, links = ["
-					+ "{ relType = projectOrganization_participation_hasParticipant, targetEntity = organization, expandAs = rel, symmetric = true, fields = [code,acronym,title,websiteurl,contracttype,fundingtree] }, "
-					+ "{ relType = resultProject_outcome_produces, targetEntity = result, expandAs = rel, symmetric = true, fields = [code,acronym,title,websiteurl,contracttype,fundingtree] }, "
-					+ "{ relType = projectPerson_contactPerson_hasContact, targetEntity = person, expandAs = rel, symmetric = true, fields = [code,acronym,title,websiteurl,contracttype,fundingtree] } "
-					+ "]}} ";
-
-	public static final String context = "[{id:'egi', label:'EGI', element:'context'}, "
-			+ "{id:'egi::classification', label:'EGI classification scheme', element:'category'}, "
-			+ "{id:'egi::classification::natsc', label:'Natural Sciences', element:'concept'}, "
-			+ "{id:'egi::classification::natsc::math', label:'Mathematics', element:'concept'}, "
-			+ "{id:'egi::classification::natsc::math::applied', label:'Applied Mathematics', element:'concept'}, "
-			+ "{id:'egi::classification::natsc::math::pure', label:'Pure Mathematics', element:'concept'}, "
-			+ "{id:'egi::classification::natsc::math::stats', label:'Statistics and Probability', element:'concept'}, "
-			+ "{id:'egi::classification::natsc::math::other', label:'Other', element:'concept'}, "
-			+ "{id:'egi::classification::natsc::cis', label:'Computer and Information sciences', element:'concept'}, "
-			+ "{id:'egi::classification::natsc::cis::ai', label:'Artificial Intelligence', element:'concept'}, "
-			+ "{id:'egi::classification::natsc::cis::bio', label:'Bioinformatics', element:'concept'}, "
-			+ "{id:'egi::classification::natsc::cis::cs', label:'Computer sciences', element:'concept'}, "
-			+ "{id:'egi::classification::natsc::cis::is', label:'Information science', element:'concept'}, "
-			+ "{id:'egi::classification::natsc::cis::other', label:'Other', element:'concept'}, "
-			+ "{id:'egi::classification::physc', label:'Physical Sciences', element:'concept'}, "
-			+ "{id:'egi::vo', label:'EGI Virtual Organizations', element:'category'}, " + "{id:'egi::vo::alice', label:'alice', element:'concept'}, "
-			+ "{id:'egi::vo::ams', label:'ams', element:'concept'}, " + "{id:'egi::vo::ams02', label:'ams02.cern.ch', element:'concept'}, "
-			+ "{id:'egi::projects', label:'EGI Projects', element:'category'}, " + "{id:'egi::projects::wenmr', label:'WeNMR', element:'concept'}]";
-
-	@Test
-	public void testParse() {
-		final IndexConfig conf = IndexConfig.load(config);
-		assertNotNull(conf);
-	}
-
-	@Test
-	public void testLoadEntityConfTable() {
-		final IndexConfig conf = IndexConfig.load(config);
-
-		final EntityConfigTable map = conf.getConfigMap();
-		assertNotNull(map);
-		assertTrue(map.size() > 0);
-		for (final Type t : Type.values()) {
-			assertTrue(map.keySet().contains(t));
-			assertNotNull(map.get(t).getIndexDuplicates());
-			for (final LinkDescriptor ld : map.get(t).getLinks().values()) {
-				assertNotNull(ld.getFields());
-				assertNotNull(ld.getTarget());
-
-				final RelDescriptor relDescriptor = ld.getRelDescriptor();
-				assertNotNull(relDescriptor);
-				assertNotNull(relDescriptor.getIt());
-				assertFalse(relDescriptor.getIt().isEmpty());
-				assertNotNull(relDescriptor.getRelType());
-				assertNotNull(relDescriptor.getSubRelType());
-
-				// System.out.println(ld);
-			}
-		}
-	}
-
-}

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/IndexConfigTest.java
___________________________________________________________________
Deleted: svn:mime-type
## -1 +0,0 ##
-text/plain
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/RelClassesTest.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/RelClassesTest.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/index/config/RelClassesTest.java	(nonexistent)
@@ -1,23 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index.config;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertNotNull;
-
-import org.junit.Test;
-
-public class RelClassesTest {
-
-	public static final String relClassesJson = "[\"isProvidedBy:provides\",\r\n\"provides:isProvidedBy\",\r\n\"merges:isMergedIn\",\r\n\"isMergedIn:merges\",\r\n\"isSimilarTo:isSimilarTo\",\r\n\"isCoAuthorOf:isCoAuthorOf\",\r\n\"isAuthorOf:hasAuthor\",\r\n\"hasAuthor:isAuthorOf\",\r\n\"isParticipant:hasParticipant\",\r\n\"hasParticipant:isParticipant\",\r\n\"isProducedBy:produces\",\r\n\"produces:isProducedBy\",\r\n\"hasAmongTopNSimilarDocuments:isAmongTopNSimilarDocuments\",\r\n\"isAmongTopNSimilarDocuments:hasAmongTopNSimilarDocuments\",\r\n\"isRelatedTo:isRelatedTo\",\r\n\"isContact:hasContact\",\r\n\"hasContact:isContact\"]";
-
-	@Test
-	public void testRelClasses() {
-		RelClasses relClasses = new RelClasses(relClassesJson);
-		assertNotNull(relClasses);
-
-		String providedBy = relClasses.getInverse("provides");
-		assertEquals(providedBy, "isProvidedBy");
-
-		String provides = relClasses.getInverse("isProvidedBy");
-		assertEquals(provides, "provides");
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/oai/OAIFeedMapperTest.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/oai/OAIFeedMapperTest.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/oai/OAIFeedMapperTest.java	(nonexistent)
@@ -1,156 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai;
-
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertTrue;
-
-import java.io.IOException;
-import java.text.ParseException;
-import java.util.Date;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.hadoop.mapreduce.Counter;
-import org.apache.solr.common.util.DateUtil;
-import org.dom4j.DocumentException;
-import org.junit.Before;
-import org.junit.Test;
-import org.mockito.Matchers;
-import org.mockito.Mock;
-import org.mockito.Mockito;
-import org.mockito.MockitoAnnotations;
-import org.springframework.core.io.ClassPathResource;
-import org.springframework.core.io.Resource;
-
-import com.google.common.collect.ArrayListMultimap;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Multimap;
-import com.mongodb.DBCollection;
-import com.mongodb.DBObject;
-
-import eu.dnetlib.data.mapreduce.hbase.oai.config.OAIConfiguration;
-import eu.dnetlib.data.mapreduce.hbase.oai.config.OAIConfigurationReader;
-import eu.dnetlib.data.mapreduce.hbase.oai.config.OAIConfigurationStringReader;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.MongoSetCollection;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.RecordFieldsExtractor;
-import eu.dnetlib.miscutils.datetime.DateUtils;
-
-public class OAIFeedMapperTest {
-
-	private OaiFeedMapper oaiFeedMapper;
-
-	/** File path where to find the XML configuration profile **/
-	private String pathToProfile = "eu/dnetlib/data/mapreduce/hbase/oai/config/OAIPublisherConfiguration-1.xml";
-
-	private OAIConfigurationStringReader oaiConfigurationReader;
-	private OAIConfiguration oaiConfiguration;
-
-	@Mock
-	private MongoSetCollection mongoSetCollection;
-	@Mock
-	private DBCollection discardedCollection;
-	@SuppressWarnings("rawtypes")
-	@Mock
-	private org.apache.hadoop.mapreduce.Mapper.Context context;
-	@Mock
-	private Counter counter;
-
-	private Date feedDate;
-	private String objId1 = "oai:dnet:openaire____::2fa6b215ace86e409dde3ba4b2a6b504";
-	private String goodRecord = "<?xml version=\"1.0\"?>\n<record>\n  <result xmlns:dc=\"http://purl.org/dc/elements/1.1/\"\n    xmlns:dnet=\"eu.dnetlib.miscutils.functional.xml.DnetXsltFunctions\"\n    xmlns:dr=\"http://www.driver-repository.eu/namespace/dr\"\n    xmlns:dri=\"http://www.driver-repository.eu/namespace/dri\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\n    <header>\n      <dri:objIdentifier>openaire____::2fa6b215ace86e409dde3ba4b2a6b504</dri:objIdentifier>\n      <dri:repositoryId/>\n      <dri:dateOfCollection>2013-10-09</dri:dateOfCollection>\n    </header>\n    <metadata>\n      <oaf:entity xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" \n\t\t    xmlns:oaf=\"http://namespace.openaire.eu/oaf\" \n\t\t    xsi:schemaLocation=\"http://namespace.openaire.eu/oaf http://www.openaire.eu/schema/0.1/oaf-0.1.xsd\">\n\t\t<oaf:datasource>\n\t\t\t<officialname>The Internet Journal of Orthopedic Surgery</officialname><englishname>The Internet Journal of Orthopedic Surgery</englishname><websiteurl>http://www.ispub.com/journal/the-internet-journal-of-orthopedic-surgery/</websiteurl><accessinfopackage/><namespaceprefix>issn15312968</namespaceprefix><datasourcetypeui classid=\"pubsrepository::journal\" classname=\"pubsrepository::journal\" schemeid=\"dnet:datasource_typologies\" schemename=\"dnet:datasource_typologies\"/><datasourcetype classid=\"pubsrepository::journal\" classname=\"pubsrepository::journal\" schemeid=\"dnet:datasource_typologies\" schemename=\"dnet:datasource_typologies\"/><openairecompatibility classid=\"notCompatible\" classname=\"notCompatible\" schemeid=\"dnet:compatibilityLevel\" schemename=\"dnet:compatibilityLevel\"/><latitude>0.0</latitude><longitude>0.0</longitude><subjects/><policies name=\"\" id=\"\"/><logourl/><contactemail/><dateofvalidation/><description/><odnumberofitems/><odnumberofitemsdate/><odpolicies/><odlanguages/><odcontenttypes/><releasestartdate/><releaseenddate/><missionstatementurl/><dataprovider>false</dataprovider><serviceprovider>false</serviceprovider><databaseaccesstype/><datauploadtype/><databaseaccessrestriction/><datauploadrestriction/><versioning>false</versioning><citationguidelineurl/><qualitymanagementkind/><pidsystems/><certificates/><originalId>openaire____::issn15312968</originalId><collectedfrom name=\"DOAJ-Articles\" id=\"driver______::bee53aa31dc2cbb538c10c2b65fa5824\"/><pid/><datainfo><inferred>false</inferred><deletedbyinference>false</deletedbyinference><trust>0.9</trust><inferenceprovenance/><provenanceaction classid=\"UNKNOWN\" classname=\"UNKNOWN\" schemeid=\"dnet:provenanceActions\" schemename=\"dnet:provenanceActions\"/></datainfo>\n\t\t  <rels>\n\t\t  </rels>\n\t\t  <children>\n\t\t  </children>\n\t\t</oaf:datasource>\n      </oaf:entity>\n    </metadata>\n  </result>\n</record>";
-	private String dedupedRecord = "dedupedRecord.xml";
-
-	@Before
-	public void setUp() throws Exception {
-		MockitoAnnotations.initMocks(this);
-		oaiFeedMapper = new OaiFeedMapper();
-
-		Resource resource = new ClassPathResource(pathToProfile);
-		// setting up the parser and the profile as a string
-		String configurationProfile = IOUtils.toString(resource.getInputStream());
-
-		System.out.println("oaiConfiguration:\n" + configurationProfile);
-		oaiConfigurationReader = new OAIConfigurationStringReader(configurationProfile);
-
-		oaiConfiguration = oaiConfigurationReader.getOaiConfiguration();
-
-		String feedDateString = DateUtils.now_ISO8601();
-		try {
-			feedDate = DateUtil.parseDate(feedDateString);
-		} catch (ParseException e) {
-			e.printStackTrace(System.err);
-			throw new RuntimeException(e);
-		}
-
-		oaiFeedMapper.setFeedDate(feedDate);
-		oaiFeedMapper.setMongoSetCollection(mongoSetCollection);
-		oaiFeedMapper.setOaiConfiguration(oaiConfiguration);
-		oaiFeedMapper.setOaiConfigurationReader(oaiConfigurationReader);
-		oaiFeedMapper.setDiscardedCollection(discardedCollection);
-		oaiFeedMapper.setSkipDuplicates(true);
-		oaiFeedMapper.setDuplicateXPath("//entity//datainfo/deletedbyinference[./text() = 'true']");
-
-		Mockito.when(mongoSetCollection.normalizeSetSpec(Matchers.anyString())).thenReturn("aNormalisedSetName");
-		Mockito.doNothing().when(counter).increment(Matchers.anyLong());
-		Mockito.when(context.getCounter(Matchers.anyString(), Matchers.anyString())).thenReturn(counter);
-		Mockito.when(discardedCollection.insert((DBObject) Matchers.any())).thenReturn(null);
-
-	}
-
-	@Test
-	public void testCreateBasicObject() throws DocumentException, IOException, InterruptedException {
-		RecordFieldsExtractor parser = new RecordFieldsExtractor(Lists.newArrayList(oaiConfiguration.getFieldsFor("oaf", "index", "openaire")));
-		Multimap<String, String> parsedRecord = parser.extractFields(goodRecord);
-		DBObject obj = oaiFeedMapper.createBasicObject(objId1, goodRecord, parsedRecord, context);
-		// NOTE that LAST_COLLECTION_DATE_FIELD, DATESTAMP_FIELD,UPDATED_FIELD are not set by the method we are testing, but by the caller
-		// method (handleRecord) because they values to set depend on the record status (NEW|UPDATED|UNCHANGED)
-		System.out.println(obj);
-		assertNotNull(obj);
-	}
-
-	@Test
-	public void testParseDeduplicated() throws IOException {
-		RecordFieldsExtractor parser = new RecordFieldsExtractor(Lists.newArrayList(oaiConfiguration.getFieldsFor("oaf", "index", "openaire")));
-		String dedupedRecordString = IOUtils.toString(this.getClass().getResourceAsStream(dedupedRecord));
-		parser.setSkipDuplicates(true);
-		parser.setDuplicateXPath("//*[local-name()='entity']//*[local-name()='datainfo']/*[local-name()='deletedbyinference'][./text() = 'true']");
-		Multimap<String, String> parsedRecord = parser.extractFields(dedupedRecordString);
-		assertFalse(oaiFeedMapper.checkRecordFields(parsedRecord, context, "x", dedupedRecordString));
-	}
-
-	@Test
-	public void testParseDeduplicated2() throws IOException {
-		RecordFieldsExtractor parser = new RecordFieldsExtractor(Lists.newArrayList(oaiConfiguration.getFieldsFor("oaf", "index", "openaire")));
-		String dedupedRecordString = IOUtils.toString(this.getClass().getResourceAsStream(dedupedRecord));
-		parser.setSkipDuplicates(true);
-		parser.setDuplicateXPath("//x");
-		Multimap<String, String> parsedRecord = parser.extractFields(dedupedRecordString);
-		assertTrue(oaiFeedMapper.checkRecordFields(parsedRecord, context, "x", dedupedRecordString));
-	}
-
-	@Test
-	public void testCheckRecordFieldsDeduplicated() {
-		Multimap<String, String> recordProps = ArrayListMultimap.create();
-		recordProps.put("duplicate", "true");
-		assertFalse(oaiFeedMapper.checkRecordFields(recordProps, context, objId1, goodRecord));
-	}
-
-	@Test
-	public void testCheckRecordFieldsNotDeduplicated() {
-		Multimap<String, String> recordProps = ArrayListMultimap.create();
-		recordProps.put("duplicate", "false");
-		recordProps.put(OAIConfigurationReader.ID_FIELD, "10|xxxx");
-		assertTrue(oaiFeedMapper.checkRecordFields(recordProps, context, objId1, goodRecord));
-	}
-
-	@Test
-	public void testCheckRecordFieldsEmpty() {
-		Multimap<String, String> recordProps = ArrayListMultimap.create();
-		assertFalse(oaiFeedMapper.checkRecordFields(recordProps, context, objId1, goodRecord));
-	}
-
-	@Test
-	public void testCheckRecordFieldsNull() {
-		Multimap<String, String> recordProps = null;
-		assertFalse(oaiFeedMapper.checkRecordFields(recordProps, context, objId1, goodRecord));
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfigurationParserTest.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfigurationParserTest.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfigurationParserTest.java	(nonexistent)
@@ -1,153 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai.config;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertTrue;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.commons.lang.StringUtils;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.junit.Before;
-import org.junit.Test;
-import org.springframework.core.io.ClassPathResource;
-import org.springframework.core.io.Resource;
-
-import com.google.common.collect.ArrayListMultimap;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-import com.google.common.collect.Multimap;
-
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.MDFInfo;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.PublisherField;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.SetInfo;
-
-public class OAIConfigurationParserTest {
-
-	private static final Log log = LogFactory.getLog(OAIConfigurationParserTest.class); // NOPMD by marko on 11/24/08 5:02 PM
-
-	/** Resource under test **/
-	private OAIConfigurationParser parser;
-	/** Configuration profile to parse **/
-	private String configurationProfile;
-	/** File path where to find the XML configuration profile **/
-	private String pathToProfile = "eu/dnetlib/data/mapreduce/hbase/oai/config/OAIPublisherConfiguration-1.xml";
-
-	/** Expected configuration based on the profile available at the given pathToProfile. **/
-	private OAIConfiguration expectedConfiguration;
-
-	@Before
-	public void setUp() throws Exception {
-
-		// preparing the expected configuration that we will use to perform asserts on the result of parsing
-		expectedConfiguration = new OAIConfiguration();
-		expectedConfiguration.setIdScheme("oai");
-		expectedConfiguration.setIdNamespace("dnet");
-		Map<String, SetInfo> setsMap = Maps.newHashMap();
-		setsMap.put("set1", new SetInfo("set1", "set number 1", "test set", "(set =\"aSet\")", true));
-		expectedConfiguration.setSetsMap(setsMap);
-		expectedConfiguration.setFieldNames(Lists.newArrayList("title", "objIdentifier", "set"));
-		Map<String, MDFInfo> formatsMap = Maps.newHashMap();
-		formatsMap.put("oai_dc", new MDFInfo("oai_dc", "SCHEMA", "NAMESPACE", "oai_dc", "store", "driver", "", true));
-		formatsMap.put("hope", new MDFInfo("hope", "SCHEMA", "NAMESPACE", "hope", "L", "I", "country = IT", "tdsRule", true));
-		expectedConfiguration.setMdFormatsMap(formatsMap);
-		List<PublisherField> fields = Lists.newArrayList();
-		String oaidcKey = "oai_dc-store-driver";
-		String hopeKey = "hope-L-I";
-
-		PublisherField titleField = new PublisherField();
-		titleField.setFieldName("title");
-		titleField.setRepeatable(true);
-		Multimap<String, String> titleSources = ArrayListMultimap.create();
-		titleSources.put(oaidcKey, "//*[local-name() = 'title']");
-		titleSources.put(oaidcKey, "//*[local-name() ='alternative']");
-		titleSources.put(hopeKey, "//title");
-		titleSources.put(hopeKey, "//titleinfo");
-		titleField.setSources(titleSources);
-
-		fields.add(titleField);
-
-		PublisherField objIdField = new PublisherField();
-		objIdField.setFieldName("objIdentifier");
-		objIdField.setRepeatable(false);
-		Multimap<String, String> objIdSources = ArrayListMultimap.create();
-		objIdSources.put(oaidcKey, "//*[local-name() ='objIdentifier']");
-		objIdField.setSources(objIdSources);
-
-		fields.add(objIdField);
-
-		PublisherField setField = new PublisherField();
-		setField.setFieldName("set");
-		setField.setRepeatable(true);
-		Multimap<String, String> setSources = ArrayListMultimap.create();
-		setSources.put(oaidcKey, "//*[local-name() ='repositoryId']");
-		setField.setSources(setSources);
-
-		fields.add(setField);
-
-		expectedConfiguration.setFields(fields);
-		Resource resource = new ClassPathResource(pathToProfile);
-		// setting up the parser and the profile as a string
-		parser = new OAIConfigurationParser();
-		configurationProfile = IOUtils.toString(resource.getInputStream());
-		log.info("setUp completed");
-	}
-
-	@Test
-	public void testGetConfiguration() throws IOException {
-		OAIConfiguration result = parser.getConfiguration(configurationProfile);
-		assertNotNull(result);
-		// asserts on sets
-		assertTrue(result.getSetsMap().containsKey("set1"));
-		SetInfo expected = expectedConfiguration.getSetsMap().get("set1");
-		SetInfo parsedSet = result.getSetsMap().get("set1");
-		assertEquals(expected.getQuery(), parsedSet.getQuery());
-		assertEquals(expected.getSetDescription(), parsedSet.getSetDescription());
-		assertEquals(expected.getSetName(), parsedSet.getSetName());
-		assertEquals(expected.getSetSpec(), parsedSet.getSetSpec());
-		// asserts on indices
-		assertTrue(result.getFieldNames().containsAll(expectedConfiguration.getFieldNames()));
-		assertEquals(3, result.getFieldsFor("oai_dc", "store", "driver").size());
-		assertEquals(1, result.getFieldsFor("hope", "L", "I").size());
-
-		// asserts on metadata formats
-		assertTrue(result.getMdFormatsMap().containsKey("hope"));
-		assertTrue(result.getMdFormatsMap().containsKey("oai_dc"));
-		MDFInfo expectedFormat = expectedConfiguration.getMdFormatsMap().get("hope");
-		MDFInfo parsedFormat = result.getMdFormatsMap().get("hope");
-		assertEquals(expectedFormat.getBaseQuery(), parsedFormat.getBaseQuery());
-		assertEquals(expectedFormat.getNamespace(), parsedFormat.getNamespace());
-		assertEquals(expectedFormat.getPrefix(), parsedFormat.getPrefix());
-		assertEquals(expectedFormat.getSchema(), parsedFormat.getSchema());
-
-		assertEquals(expectedFormat.getSourceFormatInterpretation(), parsedFormat.getSourceFormatInterpretation());
-		assertEquals(expectedFormat.getSourceFormatLayout(), parsedFormat.getSourceFormatLayout());
-		assertEquals(expectedFormat.getSourceFormatName(), parsedFormat.getSourceFormatName());
-		assertEquals(expectedFormat.getTransformationRuleID(), parsedFormat.getTransformationRuleID());
-
-		assertEquals(expectedConfiguration.getIdScheme(), result.getIdScheme());
-		assertEquals(expectedConfiguration.getIdNamespace(), result.getIdNamespace());
-
-		for (PublisherField pf : result.getFields()) {
-			for (Entry<String, String> e : pf.getSources().entries()) {
-				assertTrue(!StringUtils.isBlank(e.getValue()));
-				System.out.println(e.getValue());
-			}
-		}
-	}
-
-	@Test
-	public void testPrintConfiguration() throws IOException {
-		OAIConfiguration result = parser.getConfiguration(configurationProfile);
-		System.out.println(result);
-		System.out.println("\n\n");
-		for (PublisherField pf : result.getFields()) {
-			System.out.println(pf.toString());
-		}
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/util/LicenseComparatorTest.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/util/LicenseComparatorTest.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/util/LicenseComparatorTest.java	(nonexistent)
@@ -1,58 +0,0 @@
-package eu.dnetlib.data.mapreduce.util;
-
-import static org.junit.Assert.assertTrue;
-
-import org.junit.Before;
-import org.junit.Test;
-
-import eu.dnetlib.data.proto.FieldTypeProtos.Qualifier;
-
-public class LicenseComparatorTest {
-
-	private LicenseComparator lc;
-
-	@Before
-	public void setUp() {
-		lc = new LicenseComparator();
-	}
-
-	@Test
-	public void test() {
-
-		assertTrue(lc.compare(qualifier("OPEN"), qualifier("OPEN")) == 0);
-		assertTrue(lc.compare(qualifier("OPEN"), qualifier("EMBARGO")) < 0);
-		assertTrue(lc.compare(qualifier("OPEN"), qualifier("RESTRICTED")) < 0);
-		assertTrue(lc.compare(qualifier("OPEN"), qualifier("CLOSED")) < 0);
-		assertTrue(lc.compare(qualifier("OPEN"), qualifier("UNKNOWN")) < 0);
-
-		assertTrue(lc.compare(qualifier("EMBARGO"), qualifier("OPEN")) > 0);
-		assertTrue(lc.compare(qualifier("EMBARGO"), qualifier("EMBARGO")) == 0);
-		assertTrue(lc.compare(qualifier("EMBARGO"), qualifier("RESTRICTED")) < 0);
-		assertTrue(lc.compare(qualifier("EMBARGO"), qualifier("CLOSED")) < 0);
-		assertTrue(lc.compare(qualifier("EMBARGO"), qualifier("UNKNOWN")) < 0);
-
-		assertTrue(lc.compare(qualifier("RESTRICTED"), qualifier("OPEN")) > 0);
-		assertTrue(lc.compare(qualifier("RESTRICTED"), qualifier("EMBARGO")) > 0);
-		assertTrue(lc.compare(qualifier("RESTRICTED"), qualifier("RESTRICTED")) == 0);
-		assertTrue(lc.compare(qualifier("RESTRICTED"), qualifier("CLOSED")) < 0);
-		assertTrue(lc.compare(qualifier("RESTRICTED"), qualifier("UNKNOWN")) < 0);
-
-		assertTrue(lc.compare(qualifier("CLOSED"), qualifier("OPEN")) > 0);
-		assertTrue(lc.compare(qualifier("CLOSED"), qualifier("EMBARGO")) > 0);
-		assertTrue(lc.compare(qualifier("CLOSED"), qualifier("RESTRICTED")) > 0);
-		assertTrue(lc.compare(qualifier("CLOSED"), qualifier("CLOSED")) == 0);
-		assertTrue(lc.compare(qualifier("CLOSED"), qualifier("UNKNOWN")) < 0);
-
-		assertTrue(lc.compare(qualifier("UNKNOWN"), qualifier("OPEN")) > 0);
-		assertTrue(lc.compare(qualifier("UNKNOWN"), qualifier("EMBARGO")) > 0);
-		assertTrue(lc.compare(qualifier("UNKNOWN"), qualifier("RESTRICTED")) > 0);
-		assertTrue(lc.compare(qualifier("UNKNOWN"), qualifier("CLOSED")) > 0);
-		assertTrue(lc.compare(qualifier("UNKNOWN"), qualifier("UNKNOWN")) == 0);
-
-	}
-
-	private Qualifier qualifier(String id) {
-		return Qualifier.newBuilder().setClassid(id).build();
-	}
-
-}

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/util/LicenseComparatorTest.java
___________________________________________________________________
Deleted: svn:mime-type
## -1 +0,0 ##
-text/plain
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/util/XmlRecordFactoryTest.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/util/XmlRecordFactoryTest.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/mapreduce/util/XmlRecordFactoryTest.java	(nonexistent)
@@ -1,180 +0,0 @@
-package eu.dnetlib.data.mapreduce.util;
-
-import static org.junit.Assert.assertTrue;
-
-import java.util.List;
-
-import org.apache.commons.codec.binary.Base64;
-import org.junit.Before;
-import org.junit.Test;
-
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import com.google.protobuf.InvalidProtocolBufferException;
-import com.googlecode.protobuf.format.JsonFormat;
-
-import eu.dnetlib.data.mapreduce.hbase.index.config.ContextMapper;
-import eu.dnetlib.data.mapreduce.hbase.index.config.IndexConfig;
-import eu.dnetlib.data.mapreduce.hbase.index.config.IndexConfigTest;
-import eu.dnetlib.data.mapreduce.hbase.index.config.RelClasses;
-import eu.dnetlib.data.mapreduce.hbase.index.config.RelClassesTest;
-import eu.dnetlib.data.proto.KindProtos.Kind;
-import eu.dnetlib.data.proto.RelTypeProtos.RelType;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.data.transform.xml.AbstractDNetOafXsltFunctions;
-import eu.dnetlib.miscutils.functional.xml.IndentXmlString;
-
-public class XmlRecordFactoryTest {
-
-	public static final String CITATION_XML =
-			"<citations>\n  <citation>\n    <rawText>[10] M. Foret et al., Phys. Rev. B 66, 024204 (2002).</rawText>\n  </citation>\n  <citation>\n    <rawText>[11] B. Ru\175404\264e et al., Phys. Rev. Lett. 90, 095502 (2003).</rawText>\n  </citation>\n  <citation>\n    <rawText>[12] U. Buchenau et al., Phys. Rev. B 34, 5665 (1986).</rawText>\n  </citation>\n  <citation>\n    <rawText>[13] S.N. Taraskin and S.R. Elliott, J. Phys.: Condens. Mat- ter 11, A219 (1999).</rawText>\n  </citation>\n  <citation>\n    <rawText>[14] B. Hehlen et al., Phys. Rev. Lett. 84, 5355 (2000).</rawText>\n  </citation>\n  <citation>\n    <rawText>[15] N.V. Surotsev et al., J. Phys.: Condens. Matter 10, L113 (1998).</rawText>\n  </citation>\n  <citation>\n    <rawText>[16] D.A. Parshin and C. Laermans, Phys. Rev. B 63, 132203 (2001).</rawText>\n  </citation>\n  <citation>\n    <rawText>[17] V.L. Gurevich et al., Phys. Rev. B 67, 094203 (2003).</rawText>\n  </citation>\n  <citation>\n    <rawText>[18] A. Matic et al., Phys. Rev. Lett. 86, 3803 (2001).</rawText>\n  </citation>\n  <citation>\n    <rawText>[19] E. Rat et al., arXiv:cond-mat/0505558, 23 May 2005.</rawText>\n  </citation>\n  <citation>\n    <rawText>[1] R.C. Zeller and R.O. Pohl, Phys. Rev. B 4, 2029 (1971).</rawText>\n  </citation>\n  <citation>\n    <rawText>[20] C.A. Angell, J. Non-Cryst. Solids 131\20023133, 13 (1991).</rawText>\n  </citation>\n  <citation>\n    <rawText>[21] A.P. Sokolov et al., Phys. Rev. Lett. 71, 2062 (1993).</rawText>\n  </citation>\n  <citation>\n    <rawText>[22] T. Matsuo et al., Solid State Ionics 154-155, 759 (2002).</rawText>\n  </citation>\n  <citation>\n    <rawText>[23] V.K. Malinovsky et al., Europhys. Lett. 11, 43 (1990).</rawText>\n  </citation>\n  <citation>\n    <rawText>[24] J. Lor\250osch et al., J. Non-Cryst. Solids 69, 1 (1984).</rawText>\n  </citation>\n  <citation>\n    <rawText>[25] U. Buchenau, Z. Phys. B 58, 181 (1985).</rawText>\n  </citation>\n  <citation>\n    <rawText>[26] A.F. Io\175400e and A.R. Regel, Prog. Semicond. 4, 237 (1960).</rawText>\n  </citation>\n  <citation>\n    <rawText>[27] R. Dell\20031Anna et al., Phys. Rev. Lett. 80, 1236 (1998).</rawText>\n  </citation>\n  <citation>\n    <rawText>[28] D. Fioretto et al., Phys. Rev. E 59, 4470 (1999).</rawText>\n  </citation>\n  <citation>\n    <rawText>[29] U. Buchenau et al., Phys. Rev. Lett. 77, 4035 (1996).</rawText>\n  </citation>\n  <citation>\n    <rawText>[2] M. Rothenfusser et al., Phys. Rev. B 27, 5196 (1983).</rawText>\n  </citation>\n  <citation>\n    <rawText>[30] J. Mattsson et al., J. Phys.: Condens. Matter 15, S1259 (2003).</rawText>\n  </citation>\n  <citation>\n    <rawText>[31] T. Scopigno et al., Phys. Rev. Lett. 92, 025503 (2004).</rawText>\n  </citation>\n  <citation>\n    <rawText>[32] M. Foret et al., Phys. Rev. Lett. 81, 2100 (1998).</rawText>\n  </citation>\n  <citation>\n    <rawText>[33] F. Sette et al., Science 280, 1550 (1998).</rawText>\n  </citation>\n  <citation>\n    <rawText>[34] J. Wuttke et al., Phys. Rev. E 52, 4026 (1995).</rawText>\n  </citation>\n  <citation>\n    <rawText>[35] M.A. Ramos et al., Phys. Rev. Lett. 78, 82 (1997).</rawText>\n  </citation>\n  <citation>\n    <rawText>[36] G. Monaco et al., Phys. Rev. Lett. 80, 2161 (1998).</rawText>\n  </citation>\n  <citation>\n    <rawText>[37] A. T\250olle, Rep. Prog. Phys. 64, 1473 (2001).</rawText>\n  </citation>\n  <citation>\n    <rawText>[38] As the straight lines do not cross the origin, this does not 2 imply \1623 \21035 \1651 .</rawText>\n  </citation>\n  <citation>\n    <rawText>[39] A. Matic et al., Europhys. Lett. 54, 77 (2001).</rawText>\n  </citation>\n  <citation>\n    <rawText>[3] S. Hunklinger and W. Arnold, in Physical Acoustics, Vol. XII, W.P. Mason and R.N. Thurston Eds. (Academic Press, N.Y. 1976), p. 155.</rawText>\n  </citation>\n  <citation>\n    <rawText>[40] IXS data are usually not available below \1651co, mostly for experimental reasons. E.g., that the rapid onset was not evidenced in vitreous silica [27], is not indicative of its absence but rather of a low qco \21074 1 nm\210221.</rawText>\n  </citation>\n  <citation>\n    <rawText>[41] G. Ruocco et al., Phys. Rev. Lett. 83, 5583 (1999).</rawText>\n  </citation>\n  <citation>\n    <rawText>[42] D. C\1307 iplys et al., J. Physique (Paris) 42, C6-184 (1981).</rawText>\n  </citation>\n  <citation>\n    <rawText>[43] R. Vacher et al., Rev. Sci. Instrum. 51, 288 (1980).</rawText>\n  </citation>\n  <citation>\n    <rawText>[44] R. Vacher et al., arXiv:cond-mat/0505560, 23 May 2005.</rawText>\n  </citation>\n  <citation>\n    <rawText>[45] T.N. Claytor et al., Phys. Rev. B 18, 5842 (1978).</rawText>\n  </citation>\n  <citation>\n    <rawText>[46] M. Arai et al., Physica B 263-264, 268 (1999).</rawText>\n  </citation>\n  <citation>\n    <rawText>[4] R. Vacher et al., J. Non-Cryst. Solids 45, 397 (1981); T.C. Zhu et al., Phys. Rev. B 44, 4281 (1991).</rawText>\n  </citation>\n  <citation>\n    <rawText>[5] J.E. Graebner et al., Phys. Rev. B 34, 5696 (1986).</rawText>\n  </citation>\n  <citation>\n    <rawText>[6] E. Duval and A. Mermet, Phys. Rev. B 58, 8159 (1998).</rawText>\n  </citation>\n  <citation>\n    <rawText>[7] A. Matic et al., Phys. Rev. Lett. 93, 145502 (2004).</rawText>\n  </citation>\n  <citation>\n    <rawText>[8] Often alluded to, e.g. in the Encyclopedia of Materials: Science and Technology, K.H.J. Buschow et al., Eds., Vol. 1 (Elsevier, Oxford, 2001), articles by S.R. Elliott on pp. 171-174 and U. Buchenau on pp. 212-215.</rawText>\n  </citation>\n  <citation>\n    <rawText>[9] E. Rat et al., Phys. Rev. Lett. 83, 1355 (1999).</rawText>\n  </citation>\n</citations>";
-
-	public static final String STATISTICS_JSON =
-			"[{ \"citationsPerYear\": \"many\", \"anotherCoolStatistic\": \"WoW\", \"nestedStat\": { \"firstNestedStat\" : \"value 1\", \"secondNestedStat\" : \"value 2\"}, \"listingStat\" : [ \"one\", \"two\" ] }]";
-
-	public static final String SCHEMA_LOCATION = "http://namespace.openaire.eu/oaf http://www.openaire.eu/schema/0.2/oaf-0.2.xsd";
-
-	private XmlRecordFactory builder;
-
-	@Before
-	public void setUp() throws Exception {
-		builder =
-				new XmlRecordFactory(IndexConfig.load(IndexConfigTest.config).getConfigMap(),
-						ContextMapper.fromXml(eu.dnetlib.data.mapreduce.hbase.index.config.Context.xml), RelClasses.fromJSon(RelClassesTest.relClassesJson),
-						SCHEMA_LOCATION, true, false, false);
-	}
-
-	@Test
-	public void testJsonProtobuf() {
-		final OafDecoder decoder = OafTest.embed(OafTest.getResult("id"), Kind.entity);
-		final String json = JsonFormat.printToString(decoder.getOaf());
-		System.out.println(json);
-		System.out.println("json size: " + json.length());
-		System.out.println("binary size: " + decoder.getOaf().toByteArray().length);
-
-		final String base64String = Base64.encodeBase64String(decoder.getOaf().toByteArray());
-		System.out.println("base64 size: " + base64String.length());
-
-		System.out.println("decoded " + JsonFormat.printToString(OafDecoder.decode(Base64.decodeBase64(base64String)).getOaf()));
-	}
-
-	@Test
-	public void testProjectFP7() throws InvalidProtocolBufferException {
-
-		final String projectId = "ec::200121";
-		final String orgId = "20|organizationId";
-
-		builder.setMainEntity(OafTest.embed(OafTest.getProjectFP7(projectId, "SP3"), Kind.entity, false, false, "", "corda"));
-		builder.addRelation(Type.organization, OafTest.embed(OafTest.getProjectOrganization(orgId, projectId, "isParticipant"), Kind.relation));
-		builder.addRelation(Type.organization, OafTest.embed(OafTest.getProjectPerson("30|personId", projectId, "isContact"), Kind.relation));
-		final String xml = builder.build();
-
-		System.out.println(new IndentXmlString().evaluate(xml));
-	}
-
-	@Test
-	public void testOrganization() throws InvalidProtocolBufferException {
-
-		final String projectId = "ec::200121";
-		final String orgIdRoot = "20|organizationIdRoot";
-		final String orgIdDup = "20|organizationIdDup";
-
-		builder.setMainEntity(OafTest.embed(OafTest.getOrganization(orgIdRoot), Kind.entity, false, false, "", "corda"));
-		builder.addRelation(Type.project, OafTest.embed(OafTest.getProjectOrganization(projectId, orgIdRoot, "hasParticipant"), Kind.relation));
-		builder.addRelation(Type.datasource, OafTest.embed(OafTest.getDatasourceOrganization("10|datasourceId", orgIdRoot, "provides"), Kind.relation));
-
-		builder.addChild(Type.organization, OafTest.embed(OafTest.getOrganizationOrganization(orgIdDup, orgIdRoot, "isMergedIn"), Kind.relation));
-
-		final String xml = builder.build();
-
-		System.out.println(new IndentXmlString().evaluate(xml));
-	}
-
-	@Test
-	public void testResultFP7() throws InvalidProtocolBufferException {
-		final String resultId = "50|WOS:0001";
-		final String projectId1 = "40|ec::200121";
-		final String projectId2 = "40|ec::200122";
-		builder.setMainEntity(OafTest.embed(OafTest.getResult(resultId), Kind.entity, false, false, "", "pubmed"));
-		builder.addRelation(Type.person, OafTest.embed(OafTest.getPersonResult("30|WOS:0001_name", resultId, "1", "isAuthorOf"), Kind.relation));
-		builder.addRelation(Type.project,
-				OafTest.embed(OafTest.getResultProject(projectId1, resultId, OafTest.getProjectFP7(projectId1, "SP3"), "produces"), Kind.relation));
-		builder.addRelation(Type.project,
-				OafTest.embed(OafTest.getResultProject(projectId2, resultId, OafTest.getProjectFP7(projectId2, "SP2"), "produces"), Kind.relation));
-		builder.addRelation(Type.result,
-				OafTest.embed(OafTest.getSimilarityRel("50|WOS:0002", resultId, OafTest.getResult("50|WOS:0002"), "isAmongTopNSimilarDocuments"),
-						Kind.relation));
-		builder.addChild(Type.result, OafTest.embed(OafTest.getDedupRel("50|anotherResultId", resultId, RelType.resultResult, "isMergedIn"), Kind.relation));
-		// System.err.println(builder.toString());
-
-		System.out.println(new IndentXmlString().evaluate(builder.build()));
-		// System.out.println(builder.build());
-	}
-
-	@Test
-	public void testResultMerged() throws InvalidProtocolBufferException {
-		final String resultId = "50|WOS:0001";
-		final String similarResultId = "50|WOS:0002";
-		final String projectId1 = "40|ec::200121";
-		final String projectId2 = "40|ec::999999";
-		builder.setMainEntity(OafTest.embed(OafTest.getResult(resultId), Kind.entity, false, true, "dedup", "pubmed"));
-		builder.addRelation(Type.person, OafTest.embed(OafTest.getPersonResult("50|WOS:0001_name", resultId, "1", "isAuthorOf"), Kind.relation));
-		builder.addRelation(Type.person, OafTest.embed(OafTest.getPersonResult("50|WOS:0002_name", resultId, "2", "isAuthorOf"), Kind.relation));
-		builder.addRelation(Type.project,
-				OafTest.embed(OafTest.getResultProject(projectId1, resultId, OafTest.getProjectFP7(projectId1, "SP3"), "produces"), Kind.relation));
-		builder.addRelation(Type.project,
-				OafTest.embed(OafTest.getResultProject(projectId2, resultId, OafTest.getProjectFP7(projectId2, "SP3"), "produces"), Kind.relation));
-
-		builder.addRelation(Type.result, OafTest.embed(
-				OafTest.getSimilarityRel(similarResultId, resultId, OafTest.getResult(similarResultId), "isAmongTopNSimilarDocuments"), Kind.relation));
-
-		builder.addChild(Type.result, OafTest.embed(OafTest.getDedupRel("50|anotherResultId", resultId, RelType.resultResult, "merges"), Kind.relation));
-		final String xml = builder.build();
-
-		System.out.println(new IndentXmlString().evaluate(xml));
-	}
-
-	@Test
-	public void testDatasource() throws InvalidProtocolBufferException {
-		final String datasourceId = "10|datasourceId";
-		final String orgId = "20|organizationId";
-
-		builder.setMainEntity(OafTest.embed(OafTest.getDatasource(datasourceId), Kind.entity, false, false, "", "opendoar"));
-		builder.addRelation(Type.organization, OafTest.embed(OafTest.getDatasourceOrganization(orgId, datasourceId, "isProvidedBy"), Kind.relation));
-		final String xml = builder.build();
-
-		System.out.println(new IndentXmlString().evaluate(xml));
-	}
-
-	@Test
-	public void testProjectWT() throws InvalidProtocolBufferException {
-		final String projectId = "ec::200121";
-		builder.setMainEntity(OafTest.embed(OafTest.getProjectWT(), Kind.entity, false, false, "", "wellcometrust"));
-		builder.addChild(Type.organization, OafTest.embed(OafTest.getProjectOrganization(projectId, "20|organizationId", "isParticipant"), Kind.relation));
-		final String xml = builder.build();
-
-		System.out.println(new IndentXmlString().evaluate(xml));
-	}
-
-	@Test
-	public void testResultWT() throws InvalidProtocolBufferException {
-		final String resultId = "50|WOS:0001";
-		builder.setMainEntity(OafTest.embed(OafTest.getResult(resultId), Kind.entity, false, false, "", "arxiv"));
-		builder.addRelation(Type.person, OafTest.embed(OafTest.getPersonResult("50|WOS:0001_name", resultId, "1", "isAuthorOf"), Kind.relation));
-		builder.addRelation(Type.project, OafTest.embed(OafTest.getResultProject("40|wt::087536", resultId, OafTest.getProjectWT(), "produces"), Kind.relation));
-		builder.addChild(Type.result, OafTest.embed(OafTest.getDedupRel("50|anotherResultId", resultId, RelType.resultResult, "merges"), Kind.relation));
-		final String xml = builder.build();
-
-		System.out.println(new IndentXmlString().evaluate(xml));
-	}
-
-	@Test
-	public void testUrlFilter() throws InvalidProtocolBufferException {
-
-		final List<String> filtered =
-				Lists.newArrayList(Iterables.filter(Lists.newArrayList("http://www.google.com", "www.google.com"), AbstractDNetOafXsltFunctions.urlFilter));
-
-		assertTrue(filtered.size() == 1);
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/transform/XsltRowTransformerFactoryTest.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/transform/XsltRowTransformerFactoryTest.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/java/eu/dnetlib/data/transform/XsltRowTransformerFactoryTest.java	(nonexistent)
@@ -1,496 +0,0 @@
-package eu.dnetlib.data.transform;
-
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertTrue;
-
-import java.io.BufferedReader;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.InputStreamReader;
-import java.io.UnsupportedEncodingException;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-import java.util.zip.GZIPInputStream;
-
-import javax.xml.transform.TransformerConfigurationException;
-import javax.xml.transform.TransformerFactoryConfigurationError;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.commons.lang.StringUtils;
-import org.dom4j.DocumentException;
-import org.json.JSONObject;
-import org.junit.Before;
-import org.junit.Test;
-
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-import com.google.protobuf.InvalidProtocolBufferException;
-
-import eu.dnetlib.common.rmi.UnimplementedException;
-import eu.dnetlib.data.mapreduce.hbase.index.config.ContextMapper;
-import eu.dnetlib.data.mapreduce.hbase.index.config.EntityConfigTable;
-import eu.dnetlib.data.mapreduce.hbase.index.config.IndexConfig;
-import eu.dnetlib.data.mapreduce.hbase.index.config.IndexConfigTest;
-import eu.dnetlib.data.mapreduce.hbase.index.config.LinkDescriptor;
-import eu.dnetlib.data.mapreduce.hbase.index.config.RelClasses;
-import eu.dnetlib.data.mapreduce.hbase.index.config.RelClassesTest;
-import eu.dnetlib.data.mapreduce.util.OafDecoder;
-import eu.dnetlib.data.mapreduce.util.OafRowKeyDecoder;
-import eu.dnetlib.data.mapreduce.util.RelDescriptor;
-import eu.dnetlib.data.mapreduce.util.XmlRecordFactory;
-import eu.dnetlib.data.mapreduce.util.XmlRecordFactoryTest;
-import eu.dnetlib.data.proto.KindProtos.Kind;
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.proto.OafProtos.OafRel;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.miscutils.functional.xml.IndentXmlString;
-
-public class XsltRowTransformerFactoryTest {
-
-	private XsltRowTransformerFactory factory;
-
-	private EntityConfigTable entityConfigTable;
-
-	@Before
-	public void setUp() throws Exception {
-		factory = new XsltRowTransformerFactory();
-		entityConfigTable = IndexConfig.load(IndexConfigTest.config).getConfigMap();
-	}
-
-	@Test
-	public void testDatasource() throws Exception {
-
-		doTest(load("datasources_2_hbase.xsl"), load("datasource.xml"));
-	}
-
-	@Test
-	public void testParseClaim() throws Exception {
-
-		doTest(load("dmf_2_hbase.xsl"), load("recordClaim.xml"));
-	}
-
-	@Test
-	public void testParseProjectCorda() throws Exception {
-
-		doTest(load("projects_2_hbase.xsl"), load("projectRecordCorda.xml"));
-	}
-
-	@Test
-	public void testParseProjectFCT() throws Exception {
-
-		doTest(load("projects_2_hbase.xsl"), load("projectRecordFCT.xml"));
-	}
-
-	@Test
-	public void testParseDmf() throws Exception {
-
-		doTest(load("dmf_2_hbase.xsl"), load("record.xml"));
-	}
-
-	@Test
-	public void testParseDatacite() throws Exception {
-
-		doTest(load("datacite_2_hbase.xsl"), load("recordDatacite.xml"));
-	}
-
-	@Test
-	public void testParseDatacite2() throws Exception {
-
-		doTest(load("datacite_2_hbase.xsl"), load("recordDatacite2.xml"));
-	}
-
-	@Test
-	public void testParsePangaeOAF() throws Exception {
-
-		doTest(load("dmf_2_hbase.xsl"), load("pangaeOAF.xml"));
-	}
-
-	@Test
-	public void testParsePangaeODF() throws Exception {
-
-		doTest(load("datacite_2_hbase.xsl"), load("pangaeODF.xml"));
-	}
-
-	@Test
-	public void testLinkPangaea() throws Exception {
-
-		final List<Row> rows = Lists.newArrayList();
-		rows.addAll(asRows(load("datacite_2_hbase.xsl"), load("pangaeODF.xml")));
-		rows.addAll(asRows(load("dmf_2_hbase.xsl"), load("pangaeOAF.xml")));
-		rows.addAll(asRows(load("projects_2_hbase.xsl"), load("projectRecordCorda.xml")));
-
-		print(mapAll(buildTable(rows)));
-	}
-
-	@Test
-	public void testLinkCorda() throws Exception {
-
-		final List<Row> rows = Lists.newArrayList();
-		rows.addAll(asRows(load("projects_2_hbase.xsl"), load("projectRecordCorda.xml")));
-		rows.addAll(asRows(load("dmf_2_hbase.xsl"), load("recordCorda.xml")));
-
-		print(mapAll(buildTable(rows)));
-	}
-
-	@Test
-	public void testLinkFCT() throws Exception {
-
-		final List<Row> rows = Lists.newArrayList();
-		rows.addAll(asRows(load("projects_2_hbase.xsl"), load("projectRecordFCT.xml")));
-		rows.addAll(asRows(load("dmf_2_hbase.xsl"), load("recordFCT.xml")));
-
-		print(mapAll(buildTable(rows)));
-	}
-
-	@Test
-	public void testLinkWT() throws Exception {
-
-		final List<Row> rows = Lists.newArrayList();
-		rows.addAll(asRows(load("projects_2_hbase.xsl"), load("projectRecordWT.xml")));
-		rows.addAll(asRows(load("dmf_2_hbase.xsl"), load("recordWT.xml")));
-
-		print(mapAll(buildTable(rows)));
-	}
-
-	@Test
-	public void testLinkOrganization() throws Exception {
-
-		final List<Row> rows = Lists.newArrayList();
-		rows.addAll(asRows(load("organizations_2_hbase.xsl"), load("organization.xml")));
-		rows.addAll(asRows(load("projectorganization_2_hbase.xsl"), load("project_organization.xml")));
-		rows.addAll(asRows(load("projects_2_hbase.xsl"), load("projectRecordCorda.xml")));
-
-		print(mapAll(buildTable(rows)));
-	}
-
-	@Test
-	public void testParseDoajOAF() throws Exception {
-
-		doTest(load("dmf_2_hbase.xsl"), load("doajUniversityRecord.xml"));
-	}
-
-	// @Test
-	public void testFromMongodbCompressedDump() throws Exception {
-		doTestJsonGz(load("dmf_2_hbase.xsl"), load("mdstore_cleaned.json.gz"));
-	}
-
-	private void doTest(final InputStream xsltStream, final InputStream recordStream) throws Exception {
-		try {
-			final List<Row> rows = asRows(xsltStream, recordStream);
-
-			System.out.println(rows);
-
-			final Map<String, Map<String, Map<String, byte[]>>> table = buildTable(rows);
-
-			// System.out.println("\n" + table.toString());
-
-			final Map<String, XmlRecordFactory> builders = mapAll(table);
-
-			print(builders);
-		} catch (final InvalidProtocolBufferException e) {
-			throw new Exception(e);
-		} catch (final TransformerConfigurationException e) {
-			throw new Exception(e);
-		} catch (final TransformerFactoryConfigurationError e) {
-			throw new Exception(e);
-		} catch (final DocumentException e) {
-			throw new Exception(e);
-		}
-	}
-
-	private void doTestJsonGz(final InputStream xsltStream, final InputStream recordStream) throws Exception {
-
-		final Iterator<List<Row>> rowsIterator = asRowsJsonGzip(xsltStream, recordStream);
-
-		int i = 0;
-		while (rowsIterator.hasNext()) {
-			final List<Row> rows = rowsIterator.next();
-			i++;
-
-			if ((i % 10000) == 0) {
-				System.out.println(i);
-			}
-
-			final Map<String, Map<String, Map<String, byte[]>>> table = buildTableDoaj(rows);
-
-			for (final Map<String, Map<String, byte[]>> m : table.values()) {
-				for (final Map<String, byte[]> mv : m.values()) {
-					for (final byte[] v : mv.values()) {
-						final OafDecoder d = OafDecoder.decode(v);
-						assertNotNull(d);
-						assertNotNull(d.getOaf());
-
-						switch (d.getKind()) {
-						case entity:
-							assertNotNull(d.getMetadata());
-							if (d.getOaf().getEntity().getType().equals(Type.result)) {
-								System.out.println(d.getOaf());
-							}
-							break;
-						case relation:
-							assertNotNull(d.getRel());
-							break;
-						default:
-							break;
-						}
-					}
-				}
-			}
-		}
-	}
-
-	private List<Row> asRows(final InputStream xsltStream, final InputStream recordStream) throws Exception {
-		final String xslt = IOUtils.toString(xsltStream);
-		final XsltRowTransformer transformer = factory.getTransformer(xslt);
-		assertNotNull(transformer);
-
-		final String record = IOUtils.toString(recordStream);
-		final List<Row> rows = transformer.apply(record);
-
-		assertNotNull(rows);
-		assertFalse(rows.isEmpty());
-		return rows;
-	}
-
-	private Iterator<List<Row>> asRowsJsonGzip(final InputStream xsltStream, final InputStream recordStreamJsonGzip) throws Exception {
-		final String xslt = IOUtils.toString(xsltStream);
-		final XsltRowTransformer transformer = factory.getTransformer(xslt);
-		assertNotNull(transformer);
-		assertNotNull(recordStreamJsonGzip);
-
-		final GZIPInputStream stream = new GZIPInputStream(recordStreamJsonGzip);
-		assertNotNull(stream);
-		final BufferedReader inStream = new BufferedReader(new InputStreamReader(stream));
-		assertNotNull(inStream);
-		return new Iterator<List<Row>>() {
-
-			String jsonRecord = null;
-
-			@Override
-			public boolean hasNext() {
-				try {
-					return (jsonRecord = inStream.readLine()) != null;
-				} catch (final IOException e) {
-					throw new RuntimeException(e);
-				}
-			}
-
-			@Override
-			public List<Row> next() {
-
-				final JSONObject jsonObj = new JSONObject(jsonRecord);
-				final String body = jsonObj.getString("body");
-				try {
-					assertTrue(StringUtils.isNotBlank(body));
-					// System.out.println(body);
-					final List<Row> rows = transformer.apply(body);
-					assertNotNull(rows);
-					assertFalse(rows.isEmpty());
-					return rows;
-				} catch (final Throwable e) {
-					System.err.println("error transforming document: " + body);
-					throw new RuntimeException(e);
-				}
-			}
-
-			@Override
-			public void remove() {
-				throw new UnimplementedException();
-			}
-
-		};
-
-	}
-
-	private Map<String, Map<String, Map<String, byte[]>>> buildTableDoaj(final List<Row> rows) throws UnsupportedEncodingException {
-		final Map<String, Map<String, Map<String, byte[]>>> table = Maps.newHashMap();
-
-		for (final Row row : rows) {
-			final String rowKey = row.getKey();
-			final String cf = row.getColumnFamily();
-			if (!table.containsKey(rowKey)) {
-				table.put(rowKey, new HashMap<String, Map<String, byte[]>>());
-			}
-			if (!table.get(rowKey).containsKey(cf)) {
-				table.get(rowKey).put(row.getColumnFamily(), new HashMap<String, byte[]>());
-			}
-			for (final Column<String, byte[]> c : row.getColumns()) {
-				// System.out.println(String.format("ADDING K:%s CF:%s Q:%s", rowKey, cf, c.getName()));
-				table.get(rowKey).get(cf).put(c.getName(), c.getValue());
-				if (cf.equals("result") && c.getName().equals("body")) {
-					// System.out.println(String.format("ADDING K:%s CF:%s Q:%s", rowKey, cf, c.getName()));
-					assertTrue(StringUtils.isNotBlank(new String(c.getValue(), "UTF-8")));
-				}
-			}
-		}
-		return table;
-
-	}
-
-	private Map<String, Map<String, Map<String, byte[]>>> buildTable(final List<Row> rows) throws UnsupportedEncodingException {
-		final Map<String, Map<String, Map<String, byte[]>>> table = Maps.newHashMap();
-
-		for (final Row row : rows) {
-			final String rowKey = row.getKey();
-			final String cf = row.getColumnFamily();
-			if (!table.containsKey(rowKey)) {
-				table.put(rowKey, new HashMap<String, Map<String, byte[]>>());
-			}
-			if (!table.get(rowKey).containsKey(cf)) {
-				table.get(rowKey).put(row.getColumnFamily(), new HashMap<String, byte[]>());
-			}
-			for (final Column<String, byte[]> c : row.getColumns()) {
-				System.out.println(String.format("ADDING K:%s CF:%s Q:%s", rowKey, cf, c.getName()));
-				table.get(rowKey).get(cf).put(c.getName(), c.getValue());
-				if (cf.equals("result") && c.getName().equals("body")) {
-					final String theBody = new String(c.getValue(), "UTF-8");
-					assertTrue(StringUtils.isNotBlank(theBody));
-					// System.out.println(theBody);
-				}
-			}
-		}
-		return table;
-
-	}
-
-	private Map<String, XmlRecordFactory> mapAll(final Map<String, Map<String, Map<String, byte[]>>> table) throws Exception {
-
-		final Map<String, XmlRecordFactory> builders = Maps.newHashMap();
-		for (final Entry<String, Map<String, Map<String, byte[]>>> e : table.entrySet()) {
-			map(builders, e.getKey(), e.getValue());
-		}
-		return builders;
-	}
-
-	// private Map<String, XmlRecordFactory> mapResultsOnly(final Map<String, Map<String, Map<String, byte[]>>> table) throws Exception {
-	//
-	// final Map<String, XmlRecordFactory> builders = Maps.newHashMap();
-	// for (final Entry<String, Map<String, Map<String, byte[]>>> e : table.entrySet()) {
-	// final Type type = OafRowKeyDecoder.decode(e.getKey()).getType();
-	// if (type == Type.result) {
-	// map(builders, e.getKey(), e.getValue());
-	// }
-	// }
-	// return builders;
-	// }
-
-	private void map(final Map<String, XmlRecordFactory> builders, final String rowKey, final Map<String, Map<String, byte[]>> row) throws Exception {
-
-		final Type type = OafRowKeyDecoder.decode(rowKey).getType();
-
-		final Map<String, byte[]> colEntity = row.get(type.toString());
-
-		if (colEntity == null) return;
-
-		final byte[] bodyB = colEntity.get("body");
-
-		if (bodyB != null) {
-			ensureBuilder(builders, rowKey);
-			final OafDecoder mainEntity = OafDecoder.decode(Oaf.parseFrom(bodyB));
-			builders.get(rowKey).setMainEntity(mainEntity);
-
-			for (final LinkDescriptor ld : entityConfigTable.getDescriptors(type)) {
-
-				final String it = ld.getRelDescriptor().getIt();
-				final Map<String, byte[]> cols = row.get(it);
-
-				if ((cols != null) && !cols.isEmpty()) {
-
-					for (final byte[] oafB : cols.values()) {
-						final Oaf oaf = Oaf.parseFrom(oafB);
-
-						final OafRel.Builder relBuilder = OafRel.newBuilder(oaf.getRel());
-
-						if (ld.isSymmetric()) {
-							final RelDescriptor rd = ld.getRelDescriptor();
-							relBuilder.setCachedTarget(mainEntity.getEntity()).setRelType(rd.getRelType()).setSubRelType(rd.getSubRelType());
-						}
-
-						final OafRel oafRel = relBuilder.setChild(ld.isChild()).build();
-
-						final Oaf.Builder oafBuilder = Oaf.newBuilder().setKind(Kind.relation).setTimestamp(System.currentTimeMillis());
-						oafBuilder.getRelBuilder().mergeFrom(oafRel);
-
-						final String targetId = ld.isSymmetric() ? oafRel.getTarget() : oafRel.getSource();
-						ensureBuilder(builders, targetId);
-						final OafDecoder decoder = OafDecoder.decode(oafBuilder.build());
-
-						if (ld.isChild()) {
-							builders.get(targetId).addChild(type, decoder);
-						} else {
-							builders.get(targetId).addRelation(type, decoder);
-						}
-					}
-
-				}
-			}
-		}
-
-	}
-
-	private void print(final Map<String, XmlRecordFactory> builders) {
-		for (final Entry<String, XmlRecordFactory> e : builders.entrySet()) {
-			if (e.getValue().isValid()) {
-				System.out.println(IndentXmlString.apply(e.getValue().build()));
-			} else {
-				System.out.println("invalid builder: " + e.getKey());
-			}
-		}
-	}
-
-	private void printNoIndent(final Map<String, XmlRecordFactory> builders) {
-		for (final Entry<String, XmlRecordFactory> e : builders.entrySet()) {
-			if (e.getValue().isValid()) {
-				System.out.println(e.getValue().build());
-			} else {
-				System.out.println("invalid builder: " + e.getKey());
-			}
-		}
-	}
-
-	private void printResults(final Map<String, XmlRecordFactory> builders) {
-		for (final Entry<String, XmlRecordFactory> e : builders.entrySet()) {
-			if (e.getValue().isValid() && e.getValue().getId().startsWith("50")) {
-				System.out.println(IndentXmlString.apply(e.getValue().build()));
-			} else {
-				if (e.getValue().getId().startsWith("50")) throw new RuntimeException("invalid builder for result " + e.getKey());
-				else {
-					System.out.println("invalid builder: " + e.getKey());
-				}
-			}
-		}
-
-	}
-
-	private void ensureBuilder(final Map<String, XmlRecordFactory> builders, final String rowKey) throws Exception {
-		if (!builders.containsKey(rowKey)) {
-			builders.put(rowKey, newBuilder());
-		}
-	}
-
-	private XmlRecordFactory newBuilder() throws TransformerConfigurationException, TransformerFactoryConfigurationError, DocumentException {
-		return new XmlRecordFactory(entityConfigTable, ContextMapper.fromXml(eu.dnetlib.data.mapreduce.hbase.index.config.Context.xml),
-				RelClasses.fromJSon(RelClassesTest.relClassesJson), XmlRecordFactoryTest.SCHEMA_LOCATION, true, true, false);
-	}
-
-	private InputStream load(final String fileName) {
-		return getClass().getResourceAsStream(fileName);
-	}
-
-	@Test
-	public void test_template() throws Exception {
-		final String xslt = IOUtils.toString(load("dmf_2_hbase.xsl"));
-		final XsltRowTransformer transformer = factory.getTransformer(xslt);
-		assertNotNull(transformer);
-
-		final String record = IOUtils.toString(load("record.xml"));
-		final List<Row> rows = transformer.apply(record);
-
-		System.out.println(rows);
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/test/eu/dnetlib/data/mapreduce/openaire/feed/TestOpenaireRecordMapJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/test/eu/dnetlib/data/mapreduce/openaire/feed/TestOpenaireRecordMapJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/test/eu/dnetlib/data/mapreduce/openaire/feed/TestOpenaireRecordMapJob.java	(nonexistent)
@@ -1,24 +0,0 @@
-package eu.dnetlib.data.mapreduce.openaire.feed;
-
-import java.io.IOException;
-
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.util.ToolRunner;
-import org.junit.Before;
-import org.junit.Test;
-
-public class TestOpenaireRecordMapJob {
-
-	private OpenaireRecordMapJob job;
-	
-	@Before
-	public void setUp() throws IOException {
-		job = new OpenaireRecordMapJob();
-	}
-	
-	@Test
-	public void test() throws Exception {
-		ToolRunner.run(new Configuration(), job,  new String[] {});
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/hbase/dedup/blacklist/title_blacklist.txt
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/hbase/dedup/blacklist/title_blacklist.txt	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/hbase/dedup/blacklist/title_blacklist.txt	(nonexistent)
@@ -1,14 +0,0 @@
-^(Corpus Oral Dialectal \(COD\)\.).*$
-^(Kiri Karl Morgensternile).*$
-^(\[Eksliibris Aleksandr).*\]$
-^(Kiri A\. de Vignolles).*$
-^(2 kirja Karl Morgensternile).*$
-^(Pirita kloostri idaosa arheoloogilised).*$
-^(Kiri tundmatule).*$
-^(Kiri Jenaer Allgemeine Literaturzeitung toimetusele).*$
-^(Eksliibris Nikolai Birukovile).*$
-^(Eksliibris Nikolai Issakovile).*$
-^(\[Eksliibris Aleksandr).*$
-^(WHP Cruise Summary Information of section).*$
-^(Measurement of the top quark\-pair production cross section with ATLAS in pp collisions at).*$
-^(Measurement of the spin\-dependent structure function).*
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/hbase/dedup/applicationContext-mapreduce-dedup-job.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/hbase/dedup/applicationContext-mapreduce-dedup-job.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/hbase/dedup/applicationContext-mapreduce-dedup-job.xml	(nonexistent)
@@ -1,29 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<beans xmlns="http://www.springframework.org/schema/beans"
-	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
-	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
-
-	<!-- DEDUP JOBS -->
-
-	<bean id="dedupJob" class="eu.dnetlib.data.mapreduce.hbase.dedup.DedupJob"/>
-
-	
-	<bean id="dedupGrouperJob" class="eu.dnetlib.data.mapreduce.hbase.dedup.DedupGrouperJob"/>
-
-
-	<bean id="dedupFindRootsJob" class="eu.dnetlib.data.mapreduce.hbase.dedup.DedupFindRootsJob"/>
-
-	
-	<bean id="dedupBuildRootsJob" class="eu.dnetlib.data.mapreduce.hbase.dedup.DedupBuildRootsJob"/>
-
-	
-	<bean id="findDedupCandidatePersonsJob" class="eu.dnetlib.data.mapreduce.hbase.dedup.FindDedupCandidatePersonsJob"/>
-
-
-	<bean id="findPersonCoauthorsJob" class="eu.dnetlib.data.mapreduce.hbase.dedup.FindPersonCoauthorsJob"/>
-
-
-	<bean id="dedupPersonJob" class="eu.dnetlib.data.mapreduce.hbase.dedup.DedupPersonJob"/>
-
-	
-</beans>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/applicationContext-mapreduce-jobs.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/applicationContext-mapreduce-jobs.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/applicationContext-mapreduce-jobs.xml	(nonexistent)
@@ -1,39 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<beans xmlns="http://www.springframework.org/schema/beans"
-	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
-	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
-
-	<!-- JOBS -->		
-
-	<bean id="mdStoreHdfsImportJob" class="eu.dnetlib.data.mapreduce.hbase.dataimport.ImportRecordsJob"/>
-
-<!-- 	<bean id="mdStoreHdfsImportJob" class="eu.dnetlib.data.mapreduce.hbase.dataimport.HFileImportJob"/> -->
-	
-	<bean id="prepareFeedJob" class="eu.dnetlib.data.mapreduce.hbase.index.PrepareFeedJob"/>
-
-	
-	<bean id="indexFeedJob" class="eu.dnetlib.data.mapreduce.hbase.index.IndexFeedJob"/>
-
-	<bean id="documentDatabaseFeedJob" class="eu.dnetlib.data.mapreduce.hbase.index.DocumentDatabaseFeedJob"/>
-
-	<bean id="statsFeedJob" class="eu.dnetlib.data.mapreduce.hbase.stats.StatsFeedJob"/>
-	
-	
-	<bean id="mdStoreHBaseDeleteJob" class="eu.dnetlib.data.mapreduce.hbase.dataimport.DeleteRecordsJob"/>
-
-
-<!-- 	<bean id="hbaseTestJob"	class="eu.dnetlib.data.mapreduce.hbase.test.HBaseTestJob"/> -->
-		
-	<bean id="hbaseResetJob" class="eu.dnetlib.data.mapreduce.hbase.reset.HBaseResetJob"/>
-
-
-	<bean id="hbaseDeleteActionsJob" class="eu.dnetlib.data.mapreduce.hbase.actions.DeleteActionsJob"/>
-
-
-	<bean id="hbaseCommitActionsJob" class="eu.dnetlib.data.mapreduce.hbase.actions.CommitActionsJob"/>
-
-		
-	<bean id="hbaseCopytableJob" class="eu.dnetlib.data.mapreduce.hbase.dataexport.HBaseCopyTableJob"/>
-
-
-</beans>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/childresult.st
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/childresult.st	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/childresult.st	(nonexistent)
@@ -1,4 +0,0 @@
-<result>
-    <dri:objIdentifier>$objIdentifier$</dri:objIdentifier>
-	$metadata:{$it$}$
-</result>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/rel.st
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/rel.st	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/rel.st	(nonexistent)
@@ -1,4 +0,0 @@
-<rel inferred="$inferred$" trust="$trust$" inferenceprovenance="$inferenceprovenance$" provenanceaction="$provenanceaction$">
-  <to class="$class$" scheme="$scheme$" type="$type$">$objIdentifier$</to>
-  $metadata:{$it$}$
-</rel>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/child.st
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/child.st	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/child.st	(nonexistent)
@@ -1,3 +0,0 @@
-<$name$$if(hasId)$ objidentifier="$id$"$else$$endif$>
-	$metadata:{$it$}$
-</$name$>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/instance.st
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/instance.st	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/instance.st	(nonexistent)
@@ -1,4 +0,0 @@
-<instance id="$instanceId$">
-    $metadata:{$it$}$
-	$webresources:{$it$}$
-</instance>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/webresource.st
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/webresource.st	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/webresource.st	(nonexistent)
@@ -1,3 +0,0 @@
-<webresource>
-  <url>$identifier$</url>
-</webresource>
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/entity.st
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/entity.st	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/entity.st	(nonexistent)
@@ -1,10 +0,0 @@
-<oaf:$name$>
-	$metadata:{$it$}$
-  <rels>
-    $rels:{$it$}$
-  </rels>
-  <children>
-	$children:{$it$}$
-  </children>
-</oaf:$name$>
-$inference:{$it$}$
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/record.st
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/record.st	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/eu/dnetlib/data/mapreduce/util/record.st	(nonexistent)
@@ -1,19 +0,0 @@
-<?xml version="1.0"?>
-<record>
-  <result xmlns:dri="http://www.driver-repository.eu/namespace/dri" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
-    <header>
-      <dri:objIdentifier>$id$</dri:objIdentifier>
-      <dri:dateOfCollection>$dateofcollection$</dri:dateOfCollection>
-      <counters>
-	  $counters:{$it$}$
-	  </counters>
-    </header>
-    <metadata>
-      <oaf:entity xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
-		    xmlns:oaf="http://namespace.openaire.eu/oaf" 
-		    xsi:schemaLocation="http://namespace.openaire.eu/oaf $schemaLocation$">
-		$it$
-      </oaf:entity>
-    </metadata>
-  </result>
-</record>
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/META-INF/services/javax.xml.transform.TransformerFactory
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/META-INF/services/javax.xml.transform.TransformerFactory	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/resources/META-INF/services/javax.xml.transform.TransformerFactory	(nonexistent)
@@ -1 +0,0 @@
-net.sf.saxon.TransformerFactoryImpl
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/es/ElasticsearchFeedMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/es/ElasticsearchFeedMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/es/ElasticsearchFeedMapper.java	(nonexistent)
@@ -1,54 +0,0 @@
-package eu.dnetlib.data.mapreduce.es;
-
-import java.io.IOException;
-
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.BytesWritable;
-import org.apache.hadoop.io.NullWritable;
-
-import com.google.protobuf.InvalidProtocolBufferException;
-import com.googlecode.protobuf.format.JsonFormat;
-
-import eu.dnetlib.data.mapreduce.util.OafRowKeyDecoder;
-import eu.dnetlib.data.mapreduce.util.UpdateMerger;
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-
-public class ElasticsearchFeedMapper extends TableMapper<NullWritable, BytesWritable> {
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable keyIn, final Result value, final Context context) throws IOException, InterruptedException {
-
-		final OafRowKeyDecoder keyDecoder = OafRowKeyDecoder.decode(keyIn.copyBytes());
-
-		final Oaf oaf = mergeUpdates(value, context, keyDecoder.getType(), keyDecoder);
-
-		if (isValid(oaf)) {
-
-			context.write(NullWritable.get(), new BytesWritable(Bytes.toBytes(JsonFormat.printToString(oaf))));
-		}
-	}
-
-	private Oaf mergeUpdates(final Result value, final Context context, final Type type, final OafRowKeyDecoder keyDecoder)
-			throws InvalidProtocolBufferException {
-		try {
-			return UpdateMerger.mergeBodyUpdates(context, value.getFamilyMap(Bytes.toBytes(type.toString())));
-		} catch (final InvalidProtocolBufferException e) {
-			System.err.println(String.format("Unable to parse proto (Type: %s) in row: %s", type.toString(), keyDecoder.getKey()));
-			throw e;
-		}
-	}
-
-	private boolean isValid(final Oaf oaf) {
-		return (oaf != null) && oaf.isInitialized();
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/OptionalConfig.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/OptionalConfig.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/OptionalConfig.java	(nonexistent)
@@ -1,54 +0,0 @@
-package eu.dnetlib.data.mapreduce;
-
-import java.util.List;
-
-public class OptionalConfig {
-
-	private com.typesafe.config.Config config;
-
-	public OptionalConfig(com.typesafe.config.Config config) {
-		this.config = config;
-	}
-
-	protected Object safe(String path) {
-		return config.hasPath(path) ? config.getAnyRef(path) : null;
-	}
-
-	public Integer getInt(String path) {
-		return (Integer) safe(path);
-	}
-
-	public String getString(String path) {
-		return (String) safe(path);
-	}
-	
-	@SuppressWarnings("unchecked")
-	public <T> List<T> getList(String path) {
-		return (List<T>) safe(path);
-	}	
-
-	public Double getDouble(String path) {
-		Object safe = safe(path);
-		if (safe instanceof Integer) {
-			return Double.parseDouble(safe.toString());
-		}
-		return (Double) safe;
-	}
-
-	public Object getObject(String path) {
-		return safe(path);
-	}
-
-	public Boolean getBoolean(String path) {
-		return (Boolean) safe(path);
-	}
-	
-	public Boolean hasPath(String path) {
-		return getConfig().hasPath(path);
-	}
-
-	public com.typesafe.config.Config getConfig() {
-		return config;
-	}
-
-}

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/OptionalConfig.java
___________________________________________________________________
Deleted: svn:mime-type
## -1 +0,0 ##
-text/plain
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/OafHbaseUtils.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/OafHbaseUtils.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/OafHbaseUtils.java	(nonexistent)
@@ -1,45 +0,0 @@
-package eu.dnetlib.data.mapreduce.util;
-
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-
-import com.google.common.base.Function;
-import com.google.protobuf.InvalidProtocolBufferException;
-
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.transform.OafUtils;
-
-public class OafHbaseUtils extends OafUtils {
-
-	public static OafDecoder decode(final ImmutableBytesWritable oaf) {
-		return new OafDecoder(oaf.copyBytes());
-	}
-
-	public static Function<ImmutableBytesWritable, OafDecoder> decoder() {
-		return new Function<ImmutableBytesWritable, OafDecoder>() {
-
-			@Override
-			public OafDecoder apply(final ImmutableBytesWritable input) {
-				return OafDecoder.decode(input.copyBytes());
-			}
-		};
-	}
-
-	public static Function<ImmutableBytesWritable, Oaf> oafDecoder() {
-		return new Function<ImmutableBytesWritable, Oaf>() {
-
-			@Override
-			public Oaf apply(final ImmutableBytesWritable input) {
-				return parse(input);
-			}
-		};
-	}
-
-	public static Oaf parse(final ImmutableBytesWritable input) {
-		try {
-			return Oaf.parseFrom(input.copyBytes());
-		} catch (final InvalidProtocolBufferException e) {
-			throw new IllegalArgumentException(e);
-		}
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/LicenseComparator.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/LicenseComparator.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/LicenseComparator.java	(nonexistent)
@@ -1,46 +0,0 @@
-package eu.dnetlib.data.mapreduce.util;
-
-import java.util.Comparator;
-
-import eu.dnetlib.data.proto.FieldTypeProtos.Qualifier;
-
-public class LicenseComparator implements Comparator<Qualifier> {
-
-	@Override
-	public int compare(Qualifier left, Qualifier right) {
-
-		if (left == null && right == null) return 0;
-		if (left == null) return 1;
-		if (right == null) return -1;
-
-		String lClass = left.getClassid();
-		String rClass = right.getClassid();
-
-		if (lClass.equals(rClass)) return 0;
-
-		if (lClass.equals("OPEN")) return -1;
-		if (rClass.equals("OPEN")) return 1;
-
-		if (lClass.equals("6MONTHS")) return -1;
-		if (rClass.equals("6MONTHS")) return 1;
-
-		if (lClass.equals("12MONTHS")) return -1;
-		if (rClass.equals("12MONTHS")) return 1;
-
-		if (lClass.equals("EMBARGO")) return -1;
-		if (rClass.equals("EMBARGO")) return 1;
-
-		if (lClass.equals("RESTRICTED")) return -1;
-		if (rClass.equals("RESTRICTED")) return 1;
-
-		if (lClass.equals("CLOSED")) return -1;
-		if (rClass.equals("CLOSED")) return 1;
-
-		if (lClass.equals("UNKNOWN")) return -1;
-		if (rClass.equals("UNKNOWN")) return 1;
-
-		// Else (but unlikely), lexicographical ordering will do.
-		return Float.compare(Float.parseFloat(lClass), Float.parseFloat(rClass));
-	}
-
-}

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/LicenseComparator.java
___________________________________________________________________
Deleted: svn:mime-type
## -1 +0,0 ##
-text/plain
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/PersonResultFilter.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/PersonResultFilter.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/PersonResultFilter.java	(nonexistent)
@@ -1,24 +0,0 @@
-package eu.dnetlib.data.mapreduce.util;
-
-import com.google.common.base.Predicate;
-
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.proto.OafProtos.OafRel;
-import eu.dnetlib.data.proto.RelTypeProtos.RelType;
-
-public class PersonResultFilter implements Predicate<Oaf> {
-	private String key;
-
-	public PersonResultFilter(String key) {
-		this.key = key;
-	}
-
-	@Override
-	public boolean apply(Oaf oaf) {
-		final OafRel oafRel = oaf.getRel();
-		if (RelType.personResult.equals(oafRel.getRelType())) {
-			return !key.contains(oafRel.getSource().replaceAll("^.*\\:\\:", ""));
-		}
-		return true;
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/DedupUtils.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/DedupUtils.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/DedupUtils.java	(nonexistent)
@@ -1,190 +0,0 @@
-package eu.dnetlib.data.mapreduce.util;
-
-import java.nio.ByteBuffer;
-
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.util.Bytes;
-
-import eu.dnetlib.data.proto.DedupProtos.Dedup;
-import eu.dnetlib.data.proto.DedupSimilarityProtos.DedupSimilarity;
-import eu.dnetlib.data.proto.FieldTypeProtos.Qualifier;
-import eu.dnetlib.data.proto.OafProtos.OafRel;
-import eu.dnetlib.data.proto.OafProtos.OafRel.Builder;
-import eu.dnetlib.data.proto.OrganizationOrganizationProtos.OrganizationOrganization;
-import eu.dnetlib.data.proto.PersonPersonProtos.PersonPerson;
-import eu.dnetlib.data.proto.RelMetadataProtos.RelMetadata;
-import eu.dnetlib.data.proto.RelTypeProtos.RelType;
-import eu.dnetlib.data.proto.RelTypeProtos.SubRelType;
-import eu.dnetlib.data.proto.ResultResultProtos.ResultResult;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.pace.config.DedupConfig;
-
-public class DedupUtils {
-
-	public static final String CF_SEPARATOR = "_";
-
-	public static final String ROOT = "dedup_wf";
-
-	public static final String BODY_S = "body";
-
-	public static final byte[] BODY_B = Bytes.toBytes(BODY_S);
-
-	public static String dedupPrefix(final String dedupRun) {
-		return "|" + ROOT + "_" + dedupRun + "::";
-	}
-
-	public static String newId(final String id, final String dedupRun) {
-		if ((dedupRun == null) || (dedupRun.length() != 3)) throw new IllegalArgumentException("wrong dedupRun param");
-
-		return id.replaceFirst("\\|.*\\:\\:", dedupPrefix(dedupRun));
-	}
-
-	public static byte[] newIdBytes(final String s, final String dedupRun) {
-		return newId(s, dedupRun).getBytes();
-	}
-
-	public static byte[] newIdBytes(final ByteBuffer b, final String dedupRun) {
-		return newId(new String(b.array()), dedupRun).getBytes();
-	}
-
-	public static boolean isRoot(final String s) {
-		return s.contains(ROOT);
-	}
-
-	public static boolean isRoot(final ImmutableBytesWritable s) {
-		return isRoot(s.copyBytes());
-	}
-
-	public static boolean isRoot(final byte[] s) {
-		return isRoot(new String(s));
-	}
-
-	public static String getDedupCF_merges(final Type type) {
-		return getRelType(type) + CF_SEPARATOR + SubRelType.dedup + CF_SEPARATOR + Dedup.RelName.merges;
-	}
-
-	public static String getDedupCF_merges(final String type) {
-		return getDedupCF_merges(Type.valueOf(type));
-	}
-
-	public static byte[] getDedupCF_mergesBytes(final Type type) {
-		return Bytes.toBytes(getDedupCF_merges(type));
-	}
-
-	public static byte[] getDedupCF_mergesBytes(final String type) {
-		return getDedupCF_mergesBytes(Type.valueOf(type));
-	}
-
-	public static String getDedupCF_mergedIn(final Type type) {
-		return getRelType(type) + CF_SEPARATOR + SubRelType.dedup + CF_SEPARATOR + Dedup.RelName.isMergedIn;
-	}
-
-	public static String getDedupCF_mergedIn(final String type) {
-		return getDedupCF_mergedIn(Type.valueOf(type));
-	}
-
-	public static byte[] getDedupCF_mergedInBytes(final Type type) {
-		return Bytes.toBytes(getDedupCF_mergedIn(type));
-	}
-
-	public static byte[] getDedupCF_mergedInBytes(final String type) {
-		return getDedupCF_mergedInBytes(Type.valueOf(type));
-	}
-
-	public static String getSimilarityCF(final Type type) {
-		return getRelType(type) + CF_SEPARATOR + SubRelType.dedupSimilarity + CF_SEPARATOR + DedupSimilarity.RelName.isSimilarTo;
-	}
-
-	public static String getSimilarityCF(final String type) {
-		return getSimilarityCF(Type.valueOf(type));
-	}
-
-	public static byte[] getSimilarityCFBytes(final Type type) {
-		return Bytes.toBytes(getSimilarityCF(type));
-	}
-
-	public static byte[] getSimilarityCFBytes(final String type) {
-		return getSimilarityCFBytes(Type.valueOf(type));
-	}
-
-	public static String getRelTypeString(final Type type) {
-		return getRelType(type).toString();
-	}
-
-	public static RelType getRelType(final Type type) {
-		switch (type) {
-		case organization:
-			return RelType.organizationOrganization;
-		case person:
-			return RelType.personPerson;
-		case result:
-			return RelType.resultResult;
-		default:
-			throw new IllegalArgumentException("Deduplication not supported for entity type: " + type);
-		}
-	}
-
-	public static ColumnFamily decodeCF(final byte[] b) {
-		final String[] s = new String(b).split(CF_SEPARATOR);
-		return new DedupUtils().getCF(RelType.valueOf(s[0]), SubRelType.valueOf(s[1]));
-	}
-
-	private ColumnFamily getCF(final RelType relType, final SubRelType subRelType) {
-		return new ColumnFamily(relType, subRelType);
-	}
-
-	public static OafRel.Builder getDedup(final DedupConfig dedupConf, final String from, final String to, final Dedup.RelName relClass) {
-		final Type type = Type.valueOf(dedupConf.getWf().getEntityType());
-		final RelType relType = DedupUtils.getRelType(type);
-		final Builder oafRel =
-				OafRel.newBuilder().setRelType(relType).setSubRelType(SubRelType.dedup).setRelClass(relClass.toString()).setChild(false)
-						.setSource(new String(from)).setTarget(new String(to));
-		switch (type) {
-		case organization:
-			oafRel.setOrganizationOrganization(OrganizationOrganization.newBuilder().setDedup(
-					DedupUtils.dedup(relClass, "dnet:organization_organization_relations")));
-			break;
-		case person:
-			oafRel.setPersonPerson(PersonPerson.newBuilder().setDedup(DedupUtils.dedup(relClass, "dnet:person_person_relations")));
-			break;
-		case result:
-			oafRel.setResultResult(ResultResult.newBuilder().setDedup(DedupUtils.dedup(relClass, "dnet:result_result_relations")));
-			break;
-		default:
-			throw new IllegalArgumentException("Deduplication not supported for entity type: " + dedupConf.getWf().getEntityType());
-		}
-		return oafRel;
-	}
-
-	private static Dedup.Builder dedup(final Dedup.RelName relClass, final String scheme) {
-		return Dedup.newBuilder().setRelMetadata(
-				RelMetadata.newBuilder().setSemantics(
-						Qualifier.newBuilder().setClassid(relClass.toString()).setClassname(relClass.toString()).setSchemeid(scheme).setSchemename(scheme)));
-	}
-
-	class ColumnFamily {
-
-		private final RelType relType;
-		private final SubRelType subRelType;
-
-		public ColumnFamily(final RelType relType, final SubRelType subRelType) {
-			this.relType = relType;
-			this.subRelType = subRelType;
-		}
-
-		@Override
-		public String toString() {
-			return getRelType() + CF_SEPARATOR + getSubRelType();
-		}
-
-		public RelType getRelType() {
-			return relType;
-		}
-
-		public SubRelType getSubRelType() {
-			return subRelType;
-		}
-
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/TemplateFactory.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/TemplateFactory.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/TemplateFactory.java	(nonexistent)
@@ -1,119 +0,0 @@
-package eu.dnetlib.data.mapreduce.util;
-
-import java.io.IOException;
-import java.io.StringWriter;
-import java.util.List;
-
-import org.antlr.stringtemplate.StringTemplate;
-import org.apache.commons.io.IOUtils;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.springframework.core.io.Resource;
-
-import eu.dnetlib.data.proto.TypeProtos.Type;
-
-public class TemplateFactory {
-
-	private static final Log log = LogFactory.getLog(TemplateFactory.class); // NOPMD by marko on 11/24/08 5:02 PM
-
-	protected TemplateResources resources = new TemplateResources();
-
-	public String buildBody(final Type type, final List<String> metadata, final List<String> rels, final List<String> children, final List<String> inference) {
-
-		StringTemplate body = getTemplate(resources.getEntity());
-
-		body.setAttribute("name", type.toString());
-		body.setAttribute("metadata", metadata);
-		body.setAttribute("rels", rels);
-		body.setAttribute("children", children);
-		body.setAttribute("inference", inference);
-
-		return body.toString();
-	}
-
-	public String getChild(final String name, final String id, final List<String> metadata) {
-		StringTemplate child = getTemplate(resources.getChild());
-
-		child.setAttribute("name", name);
-		child.setAttribute("hasId", !(id == null));
-		child.setAttribute("id", id != null ? XmlRecordFactory.escapeXml(XmlRecordFactory.removePrefix(id)) : "");
-		child.setAttribute("metadata", metadata);
-
-		String string = child.toString();
-		return string;
-	}
-
-	public String buildRecord(final Type type,
-			final String objIdentifier,
-			final String dateOfCollection,
-			final String schemaLocation,
-			final String body,
-			final List<String> counters) {
-		final StringTemplate recordTemplate = getTemplate(resources.getRecord());
-
-		recordTemplate.setAttribute("id", XmlRecordFactory.escapeXml(XmlRecordFactory.removePrefix(objIdentifier)));
-		recordTemplate.setAttribute("dateofcollection", dateOfCollection);
-		recordTemplate.setAttribute("schemaLocation", schemaLocation);
-		recordTemplate.setAttribute("counters", counters);
-		recordTemplate.setAttribute("it", body);
-
-		return recordTemplate.toString();
-	}
-
-	public String getRel(final Type type,
-			final String objIdentifier,
-			final List<String> fields,
-			final String semanticclass,
-			final String semantischeme,
-			final boolean inferred,
-			final String trust,
-			final String inferenceprovenance,
-			final String provenanceaction) {
-		final StringTemplate template = getTemplate(resources.getRel());
-
-		template.setAttribute("type", type.toString());
-		template.setAttribute("objIdentifier", XmlRecordFactory.escapeXml(XmlRecordFactory.removePrefix(objIdentifier)));
-		template.setAttribute("class", semanticclass);
-		template.setAttribute("scheme", semantischeme);
-		template.setAttribute("metadata", fields);
-		template.setAttribute("inferred", inferred);
-		template.setAttribute("trust", trust);
-		template.setAttribute("inferenceprovenance", inferenceprovenance);
-		template.setAttribute("provenanceaction", provenanceaction);
-
-		return template.toString();
-	}
-
-	public String getInstance(final String resultId, final List<String> instancemetadata, final List<String> webresources) {
-		final StringTemplate template = getTemplate(resources.getInstance());
-
-		template.setAttribute("instanceId", XmlRecordFactory.escapeXml(XmlRecordFactory.removePrefix(resultId)));
-		template.setAttribute("metadata", instancemetadata);
-		template.setAttribute("webresources", webresources);
-
-		return template.toString();
-	}
-
-	public String getWebResource(final String identifier) {
-		final StringTemplate template = getTemplate(resources.getWebresource());
-
-		template.setAttribute("identifier", XmlRecordFactory.escapeXml(identifier));
-
-		return template.toString();
-	}
-
-	// ////////////////////////////////////////////////////
-
-	private StringTemplate getTemplate(final Resource res) {
-		StringWriter body = new StringWriter();
-		try {
-			IOUtils.copy(res.getInputStream(), body);
-			String string = body.toString();
-			return new StringTemplate(string);
-		} catch (IOException e) {
-			log.error("unable to get template", e);
-			throw new RuntimeException(e);
-		}
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/UpdateMerger.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/UpdateMerger.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/UpdateMerger.java	(nonexistent)
@@ -1,54 +0,0 @@
-package eu.dnetlib.data.mapreduce.util;
-
-import java.util.Collections;
-import java.util.List;
-import java.util.Map;
-
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.mapreduce.Mapper.Context;
-
-import com.google.common.collect.Lists;
-import com.google.protobuf.InvalidProtocolBufferException;
-
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.transform.OafEntityMerger;
-
-public class UpdateMerger {
-
-	private static final String UPDATE_MERGER = "UPDATE_MERGE";
-	private static final String N_MERGES = "N_MERGES";
-
-	@SuppressWarnings("rawtypes")
-	public static Oaf mergeBodyUpdates(final Context context, final Map<byte[], byte[]> map) throws InvalidProtocolBufferException {
-		final byte[] value = map.get(DedupUtils.BODY_B);
-		if (value == null) return null;
-
-		Oaf.Builder builder = Oaf.newBuilder(Oaf.parseFrom(value));
-		final List<String> keys = Lists.newArrayList();
-
-		// we fetch all the body updates
-		for (final byte[] o : map.keySet()) {
-			final String sKey = Bytes.toString(o);
-			if (sKey.startsWith("update_")) {
-				keys.add(sKey);
-			}
-		}
-		if (!keys.isEmpty()) {
-			// we merge all the sorted updates with the body
-			Collections.sort(keys);
-			for (final String k : keys) {
-				final Oaf update = Oaf.parseFrom(map.get(Bytes.toBytes(k)));
-				// System.out.println("\n\nBODY: \n" + body.build().toString());
-				// System.out.println("UPDATE: \n" + update.toString());
-				builder.mergeFrom(update);
-				// System.out.println("UDPATED BODY: \n" + body.build().toString() + "\n\n");
-			}
-
-			builder = OafEntityMerger.merge(builder);
-
-			context.getCounter(UPDATE_MERGER, N_MERGES).increment(keys.size());
-		}
-		return builder.build();
-	}
-
-}

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/UpdateMerger.java
___________________________________________________________________
Deleted: svn:mime-type
## -1 +0,0 ##
-text/plain
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/TemplateResources.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/TemplateResources.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/TemplateResources.java	(nonexistent)
@@ -1,50 +0,0 @@
-package eu.dnetlib.data.mapreduce.util;
-
-import org.springframework.core.io.ClassPathResource;
-import org.springframework.core.io.Resource;
-
-public class TemplateResources {
-	
-	private Resource record = new ClassPathResource("/eu/dnetlib/data/mapreduce/util/record.st");
-		
-	private Resource instance = new ClassPathResource("/eu/dnetlib/data/mapreduce/util/instance.st");
-	
-	private Resource rel = new ClassPathResource("/eu/dnetlib/data/mapreduce/util/rel.st");
-	
-	private Resource webresource = new ClassPathResource("/eu/dnetlib/data/mapreduce/util/webresource.st");
-	
-	private Resource child = new ClassPathResource("/eu/dnetlib/data/mapreduce/util/child.st");
-	
-	private Resource tree = new ClassPathResource("/eu/dnetlib/data/mapreduce/util/tree.st");
-	
-	private Resource entity = new ClassPathResource("/eu/dnetlib/data/mapreduce/util/entity.st");
-	
-	public Resource getEntity() {
-		return entity;
-	}	
-	
-	public Resource getRecord() {
-		return record;
-	}
-	
-	public Resource getInstance() {
-		return instance;
-	}
-
-	public Resource getRel() {
-		return rel;
-	}
-
-	public Resource getWebresource() {
-		return webresource;
-	}
-
-	public Resource getChild() {
-		return child;
-	}
-	
-	public Resource getTree() {
-		return tree;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/XmlRecordFactory.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/XmlRecordFactory.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/util/XmlRecordFactory.java	(nonexistent)
@@ -1,766 +0,0 @@
-package eu.dnetlib.data.mapreduce.util;
-
-import static eu.dnetlib.miscutils.collections.MappedCollection.listMap;
-
-import java.io.StringReader;
-import java.io.StringWriter;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-import java.util.Set;
-import java.util.StringTokenizer;
-
-import javax.xml.transform.OutputKeys;
-import javax.xml.transform.Transformer;
-import javax.xml.transform.TransformerConfigurationException;
-import javax.xml.transform.TransformerException;
-import javax.xml.transform.TransformerFactory;
-import javax.xml.transform.TransformerFactoryConfigurationError;
-import javax.xml.transform.dom.DOMSource;
-import javax.xml.transform.stream.StreamResult;
-
-import org.apache.commons.lang.StringUtils;
-import org.dom4j.Document;
-import org.dom4j.DocumentException;
-import org.dom4j.Element;
-import org.dom4j.Node;
-import org.dom4j.io.SAXReader;
-
-import com.google.common.base.Joiner;
-import com.google.common.base.Predicate;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-import com.google.common.collect.Sets;
-import com.google.protobuf.Descriptors.EnumValueDescriptor;
-import com.google.protobuf.Descriptors.FieldDescriptor;
-import com.google.protobuf.GeneratedMessage;
-import com.mycila.xmltool.XMLDoc;
-import com.mycila.xmltool.XMLTag;
-
-import eu.dnetlib.data.mapreduce.hbase.index.config.ContextDef;
-import eu.dnetlib.data.mapreduce.hbase.index.config.ContextMapper;
-import eu.dnetlib.data.mapreduce.hbase.index.config.EntityConfigTable;
-import eu.dnetlib.data.mapreduce.hbase.index.config.LinkDescriptor;
-import eu.dnetlib.data.mapreduce.hbase.index.config.RelClasses;
-import eu.dnetlib.data.proto.FieldTypeProtos.BoolField;
-import eu.dnetlib.data.proto.FieldTypeProtos.DataInfo;
-import eu.dnetlib.data.proto.FieldTypeProtos.ExtraInfo;
-import eu.dnetlib.data.proto.FieldTypeProtos.KeyValue;
-import eu.dnetlib.data.proto.FieldTypeProtos.Qualifier;
-import eu.dnetlib.data.proto.FieldTypeProtos.StringField;
-import eu.dnetlib.data.proto.FieldTypeProtos.StructuredProperty;
-import eu.dnetlib.data.proto.OafProtos.OafEntity;
-import eu.dnetlib.data.proto.OafProtos.OafRel;
-import eu.dnetlib.data.proto.PersonProtos.Person;
-import eu.dnetlib.data.proto.ProjectProtos.Project;
-import eu.dnetlib.data.proto.RelMetadataProtos.RelMetadata;
-import eu.dnetlib.data.proto.ResultProtos.Result;
-import eu.dnetlib.data.proto.ResultProtos.Result.Context;
-import eu.dnetlib.data.proto.ResultProtos.Result.ExternalReference;
-import eu.dnetlib.data.proto.ResultProtos.Result.Instance;
-import eu.dnetlib.data.proto.ResultProtos.Result.Journal;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.miscutils.functional.UnaryFunction;
-
-public class XmlRecordFactory {
-
-	// private static final Log log = LogFactory.getLog(XmlRecordFactory.class); // NOPMD by marko on 11/24/08 5:02 PM
-
-	protected Set<String> specialDatasourceTypes = Sets.newHashSet("scholarcomminfra", "infospace", "pubsrepository::mock", "entityregistry");
-
-	protected TemplateFactory templateFactory = new TemplateFactory();
-
-	protected OafDecoder mainEntity = null;
-
-	protected String key = null;
-
-	protected List<OafDecoder> relations = Lists.newLinkedList();
-	protected List<OafDecoder> children = Lists.newLinkedList();
-
-	protected EntityConfigTable entityConfigTable;
-
-	protected ContextMapper contextMapper;
-
-	protected RelClasses relClasses;
-
-	protected String schemaLocation;
-
-	protected boolean entityDefaults;
-	protected boolean relDefaults;
-	protected boolean childDefaults;
-
-	protected Set<String> contextes = Sets.newHashSet();
-
-	protected List<String> extraInfo = Lists.newArrayList();
-
-	protected Map<String, Integer> counters = Maps.newHashMap();
-
-	protected Transformer transformer;
-
-	private final Map<String, Integer> relCounters = Maps.newHashMap();
-
-	public XmlRecordFactory(final EntityConfigTable entityConfigTable, final ContextMapper contextMapper, final RelClasses relClasses,
-			final String schemaLocation, final boolean entityDefaults, final boolean relDefaults, final boolean childDefeaults)
-			throws TransformerConfigurationException, TransformerFactoryConfigurationError {
-		this.entityConfigTable = entityConfigTable;
-		this.contextMapper = contextMapper;
-		this.relClasses = relClasses;
-		this.schemaLocation = schemaLocation;
-		this.entityDefaults = entityDefaults;
-		this.relDefaults = relDefaults;
-		this.childDefaults = childDefeaults;
-
-		transformer = TransformerFactory.newInstance().newTransformer();
-		transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
-	}
-
-	public Map<String, Integer> getRelCounters() {
-		return relCounters;
-	}
-
-	public RelClasses getRelClasses() {
-		return relClasses;
-	}
-
-	public String getId() {
-		return key;
-	}
-
-	public boolean isValid() {
-		return mainEntity != null;
-	}
-
-	public void setMainEntity(final OafDecoder mainEntity) {
-		this.mainEntity = mainEntity;
-		this.key = mainEntity.decodeEntity().getId();
-	}
-
-	public void addRelation(final Type type, final OafDecoder rel) {
-		addRelOrChild(type, relations, rel);
-	}
-
-	public void addChild(final Type type, final OafDecoder child) {
-		addRelOrChild(type, children, child);
-	}
-
-	private void addRelOrChild(final Type type, final List<OafDecoder> list, final OafDecoder decoder) {
-
-		final OafRel oafRel = decoder.getOafRel();
-		final String rd = oafRel.getRelType().toString() + "_" + oafRel.getSubRelType() + "_" + relClasses.getInverse(oafRel.getRelClass());
-		final LinkDescriptor ld = entityConfigTable.getDescriptor(type, new RelDescriptor(rd));
-
-		if (getRelCounters().get(rd) == null) {
-			getRelCounters().put(rd, 0);
-		}
-
-		if (ld == null) {
-			list.add(decoder);
-			return;
-		}
-
-		if (ld.getMax() < 0) {
-			list.add(decoder);
-			return;
-		}
-
-		if (getRelCounters().get(rd) < ld.getMax()) {
-			getRelCounters().put(rd, getRelCounters().get(rd) + 1);
-			list.add(decoder);
-		}
-	}
-
-	public String build() {
-
-		final OafEntityDecoder entity = mainEntity.decodeEntity();
-		// log.info("building");
-		// log.info("main: " + mainEntity);
-		// log.info("rel:  " + relations);
-		// log.info("chi:  " + children);
-		// log.info("=============");
-
-		final Type type = entity.getType();
-		final List<String> metadata = decodeType(entity, null, entityDefaults, false);
-
-		// rels has to be processed before the contexts because they enrich the contextMap with the funding info.
-		final List<String> rels = listRelations();
-		metadata.addAll(buildContexts(type));
-		metadata.add(parseDataInfo(mainEntity));
-
-		final String body = templateFactory.buildBody(type, metadata, rels, listChildren(), extraInfo);
-
-		// log.info("record id: " + recordId);
-		return templateFactory.buildRecord(type, key, entity.getDateOfCollection(), schemaLocation, body, countersAsXml());
-	}
-
-	private String parseDataInfo(final OafDecoder decoder) {
-		final DataInfo dataInfo = decoder.getOaf().getDataInfo();
-
-		final StringBuilder sb = new StringBuilder();
-		sb.append("<datainfo>");
-		sb.append(asXmlElement("inferred", dataInfo.getInferred() + "", null, null));
-		sb.append(asXmlElement("deletedbyinference", dataInfo.getDeletedbyinference() + "", null, null));
-		sb.append(asXmlElement("trust", dataInfo.getTrust() + "", null, null));
-		sb.append(asXmlElement("inferenceprovenance", dataInfo.getInferenceprovenance() + "", null, null));
-		sb.append(asXmlElement("provenanceaction", null, dataInfo.getProvenanceaction(), null));
-		sb.append("</datainfo>");
-
-		return sb.toString();
-	}
-
-	private List<String> decodeType(final OafEntityDecoder decoder, final Set<String> filter, final boolean defaults, final boolean expandingRel) {
-
-		final List<String> metadata = Lists.newArrayList();
-		metadata.addAll(listFields(decoder.getMetadata(), filter, defaults, expandingRel));
-		metadata.addAll(listFields(decoder.getOafEntity(), filter, defaults, expandingRel));
-
-		if ((decoder.getEntity() instanceof Result) && !expandingRel) {
-			metadata.add(asXmlElement("bestlicense", "", getBestLicense(), null));
-
-			metadata.addAll(listFields(decoder.getEntity(), filter, defaults, expandingRel));
-		}
-		if ((decoder.getEntity() instanceof Person) && !expandingRel) {
-			metadata.addAll(listFields(decoder.getEntity(), filter, defaults, expandingRel));
-		}
-		if ((decoder.getEntity() instanceof Project) && !expandingRel) {
-			metadata.addAll(listFields(decoder.getEntity(), filter, defaults, expandingRel));
-		}
-
-		return metadata;
-	}
-
-	private Qualifier getBestLicense() {
-		Qualifier bestLicense = getQualifier("UNKNOWN", "not available", "dnet:access_modes");
-		final LicenseComparator lc = new LicenseComparator();
-		for (final Instance instance : ((Result) mainEntity.decodeEntity().getEntity()).getInstanceList()) {
-			if (lc.compare(bestLicense, instance.getLicence()) > 0) {
-				bestLicense = instance.getLicence();
-			}
-		}
-		return bestLicense;
-	}
-
-	public Qualifier getQualifier(final String classid, final String classname, final String schemename) {
-		return Qualifier.newBuilder().setClassid(classid).setClassname(classname).setSchemeid(schemename).setSchemename(schemename).build();
-	}
-
-	private List<String> listRelations() {
-
-		final List<String> rels = Lists.newArrayList();
-
-		for (final OafDecoder decoder : this.relations) {
-
-			final OafRel rel = decoder.getOafRel();
-			final OafEntity cachedTarget = rel.getCachedTarget();
-			final OafRelDecoder relDecoder = OafRelDecoder.decode(rel);
-
-			// if (!relDecoder.getRelType().equals(RelType.personResult) || relDecoder.getRelTargetId().equals(key)) {
-			if (relDecoder.getRelSourceId().equals(key) || relDecoder.getRelTargetId().equals(key)) {
-
-				final List<String> metadata = Lists.newArrayList();
-				final Type targetType = relDecoder.getTargetType(mainEntity.getEntity().getType());
-				final Set<String> relFilter = entityConfigTable.getFilter(targetType, relDecoder.getRelDescriptor());
-				metadata.addAll(listFields(relDecoder.getSubRel(), relFilter, false, true));
-
-				String semanticclass = "";
-				String semanticscheme = "";
-
-				final RelDescriptor relDescriptor = relDecoder.getRelDescriptor();
-
-				if ((cachedTarget != null) && cachedTarget.isInitialized()) {
-
-					final Set<String> filter = entityConfigTable.getFilter(targetType, relDescriptor);
-					metadata.addAll(decodeType(OafEntityDecoder.decode(cachedTarget), filter, relDefaults, true));
-				}
-
-				final RelMetadata relMetadata = relDecoder.getRelMetadata();
-				// debug
-				if (relMetadata == null) {
-					// System.err.println(this);
-					semanticclass = semanticscheme = "UNKNOWN";
-				} else {
-					semanticclass = relClasses.getInverse(relMetadata.getSemantics().getClassname());
-					semanticscheme = relMetadata.getSemantics().getSchemename();
-				}
-
-				incrementCounter(relDescriptor.getSubRelType().toString());
-
-				final LinkDescriptor ld = entityConfigTable.getDescriptor(relDecoder.getTargetType(mainEntity.getEntity().getType()), relDescriptor);
-
-				final String relId = (ld != null) && !ld.isSymmetric() ? relDecoder.getRelTargetId() : relDecoder.getRelSourceId();
-
-				final DataInfo info = decoder.getOaf().getDataInfo();
-
-				rels.add(templateFactory.getRel(targetType, relId, metadata, semanticclass, semanticscheme, info.getInferred(), info.getTrust(),
-						info.getInferenceprovenance(), info.getProvenanceaction().getClassid()));
-			}
-		}
-		return rels;
-	}
-
-	private List<String> listChildren() {
-
-		final List<String> children = Lists.newArrayList();
-		for (final OafDecoder decoder : this.children) {
-			final OafEntity cachedTarget = decoder.getOafRel().getCachedTarget();
-			addChildren(children, cachedTarget, decoder.getRelDescriptor());
-		}
-		final OafEntityDecoder entity = mainEntity.decodeEntity();
-		if (entity.getType().equals(Type.result)) {
-			for (final Instance instance : ((Result) entity.getEntity()).getInstanceList()) {
-				final Set<String> urlFilter = Sets.newHashSet("instancetype", "hostedby");
-				children.add(templateFactory.getInstance(instance.getHostedby().getKey(), listFields(instance, urlFilter, false, false),
-						listMap(instance.getUrlList(), new UnaryFunction<String, String>() {
-
-							@Override
-							public String evaluate(final String identifier) {
-								return templateFactory.getWebResource(identifier);
-							}
-						})));
-			}
-			for (final ExternalReference er : ((Result) entity.getEntity()).getExternalReferenceList()) {
-				// Set<String> filters = entityConfigTable.getFilter(Type.result, RelType.resultResult);
-				final List<String> fields = listFields(er, null, false, false);
-				children.add(templateFactory.getChild("externalreference", null, fields));
-			}
-		}
-
-		return children;
-	}
-
-	private void addChildren(final List<String> children, final OafEntity target, final RelDescriptor relDescriptor) {
-		final OafEntityDecoder decoder = OafEntityDecoder.decode(target);
-		incrementCounter(relDescriptor.getSubRelType().toString());
-		final Set<String> filters = entityConfigTable.getFilter(target.getType(), relDescriptor);
-		children.add(templateFactory.getChild(decoder.getType().toString(), decoder.getId(), listFields(decoder.getMetadata(), filters, childDefaults, false)));
-	}
-
-	// //////////////////////////////////
-
-	private List<String> listFields(final GeneratedMessage fields, final Set<String> filter, final boolean defaults, final boolean expandingRel) {
-
-		final List<String> metadata = Lists.newArrayList();
-
-		if (fields != null) {
-
-			final Set<String> seen = Sets.newHashSet();
-			for (final Entry<FieldDescriptor, Object> e : filterFields(fields, filter)) {
-
-				// final String name = getFieldName(e.getKey().getName());
-				final String name = e.getKey().getName();
-				seen.add(name);
-
-				addFieldValue(metadata, e.getKey(), e.getValue(), expandingRel);
-			}
-
-			if (defaults) {
-				for (final FieldDescriptor fd : fields.getDescriptorForType().getFields()) {
-					if (!seen.contains(fd.getName())) {
-						addFieldValue(metadata, fd, getDefault(fd), expandingRel);
-					}
-				}
-			}
-		}
-		return metadata;
-	}
-
-	private Object getDefault(final FieldDescriptor fd) {
-		switch (fd.getType()) {
-		case BOOL:
-			return false;
-		case BYTES:
-			return "".getBytes();
-		case MESSAGE: {
-			if (Qualifier.getDescriptor().equals(fd.getMessageType())) return defaultQualifier();
-			if (StructuredProperty.getDescriptor().equals(fd.getMessageType())) return StructuredProperty.newBuilder().setValue("")
-					.setQualifier(defaultQualifier()).build();
-			if (KeyValue.getDescriptor().equals(fd.getMessageType())) return KeyValue.newBuilder().setKey("").setValue("").build();
-			if (StringField.getDescriptor().equals(fd.getMessageType())) return StringField.newBuilder().setValue("").build();
-			if (BoolField.getDescriptor().equals(fd.getMessageType())) return BoolField.newBuilder().buildPartial();
-			return null;
-		}
-		case SFIXED32:
-		case SFIXED64:
-		case SINT32:
-		case SINT64:
-		case INT32:
-		case INT64:
-		case DOUBLE:
-		case FIXED32:
-		case FIXED64:
-		case FLOAT:
-			return 0;
-		case STRING:
-			return "";
-		default:
-			return null;
-		}
-	}
-
-	private Qualifier defaultQualifier() {
-		return Qualifier.newBuilder().setClassid("").setClassname("").setSchemeid("").setSchemename("").build();
-	}
-
-	@SuppressWarnings("unchecked")
-	private void addFieldValue(final List<String> metadata, final FieldDescriptor fd, final Object value, final boolean expandingRel) {
-		if (fd.getName().equals("dateofcollection") || fd.getName().equals("id") || /* fd.getName().equals("url") || */(value == null)) return;
-
-		if (fd.getName().equals("datasourcetype")) {
-			final String classid = ((Qualifier) value).getClassid();
-
-			final Qualifier.Builder q = Qualifier.newBuilder((Qualifier) value);
-			if (specialDatasourceTypes.contains(classid)) {
-				q.setClassid("other").setClassname("other");
-			}
-			metadata.add(asXmlElement("datasourcetypeui", "", q.build(), null));
-		}
-
-		if (fd.isRepeated() && (value instanceof List<?>)) {
-			for (final Object o : (List<Object>) value) {
-				guessType(metadata, fd, o, expandingRel);
-			}
-		} else {
-			guessType(metadata, fd, value, expandingRel);
-		}
-	}
-
-	private void guessType(final List<String> metadata, final FieldDescriptor fd, final Object o, final boolean expandingRel) {
-
-		if (fd.getType().equals(FieldDescriptor.Type.MESSAGE)) {
-
-			if (Qualifier.getDescriptor().equals(fd.getMessageType())) {
-				final Qualifier qualifier = (Qualifier) o;
-				metadata.add(asXmlElement(fd.getName(), "", qualifier, null));
-			}
-
-			if (StructuredProperty.getDescriptor().equals(fd.getMessageType())) {
-				final StructuredProperty sp = (StructuredProperty) o;
-				metadata.add(asXmlElement(fd.getName(), sp.getValue(), sp.getQualifier(), sp.hasDataInfo() ? sp.getDataInfo() : null));
-			}
-
-			if (KeyValue.getDescriptor().equals(fd.getMessageType())) {
-				final KeyValue kv = (KeyValue) o;
-				metadata.add("<" + fd.getName() + " name=\"" + escapeXml(kv.getValue()) + "\" id=\"" + escapeXml(removePrefix(kv.getKey())) + "\"/>");
-			}
-
-			if (StringField.getDescriptor().equals(fd.getMessageType())) {
-				final String fieldName = fd.getName();
-
-				if (fieldName.equals("fundingtree")) {
-					final String xmlTree = o instanceof StringField ? ((StringField) o).getValue() : o.toString();
-
-					if (expandingRel) {
-						metadata.add(getRelFundingTree(xmlTree));
-						fillContextMap(xmlTree);
-					} else {
-						metadata.add(xmlTree);
-					}
-				} else {
-					final StringField sf = (StringField) o;
-					final StringBuilder sb = new StringBuilder("<" + fd.getName());
-					if (sf.hasDataInfo()) {
-						final DataInfo dataInfo = sf.getDataInfo();
-						dataInfoAsAttributes(sb, dataInfo);
-					}
-					sb.append(">" + escapeXml(sf.getValue()) + "</" + fd.getName() + ">");
-					metadata.add(sb.toString());
-				}
-			}
-
-			if (BoolField.getDescriptor().equals(fd.getMessageType())) {
-				final BoolField bf = (BoolField) o;
-				final StringBuilder sb = new StringBuilder("<" + fd.getName());
-				if (bf.hasDataInfo()) {
-					final DataInfo dataInfo = bf.getDataInfo();
-					dataInfoAsAttributes(sb, dataInfo);
-				}
-
-				sb.append(">" + (bf.hasValue() ? bf.getValue() : "") + "</" + fd.getName() + ">");
-				metadata.add(sb.toString());
-			}
-
-			if (Journal.getDescriptor().equals(fd.getMessageType()) && (o != null)) {
-				final Journal j = (Journal) o;
-				metadata.add("<journal " + "issn=\"" + escapeXml(j.getIssnPrinted()) + "\" " + "eissn=\"" + escapeXml(j.getIssnOnline()) + "\" " + "lissn=\""
-						+ escapeXml(j.getIssnLinking()) + "\">" + escapeXml(j.getName()) + "</journal>");
-			}
-
-			if (Context.getDescriptor().equals(fd.getMessageType()) && (o != null)) {
-				contextes.add(((Result.Context) o).getId());
-			}
-
-			if (ExtraInfo.getDescriptor().equals(fd.getMessageType()) && (o != null)) {
-
-				final ExtraInfo e = (ExtraInfo) o;
-				final StringBuilder sb = new StringBuilder("<" + fd.getName() + " ");
-
-				sb.append("name=\"" + e.getName() + "\" ");
-				sb.append("typology=\"" + e.getTypology() + "\" ");
-				sb.append("provenance=\"" + e.getProvenance() + "\" ");
-				sb.append("trust=\"" + e.getTrust() + "\"");
-				sb.append(">");
-				sb.append(e.getValue());
-				sb.append("</" + fd.getName() + ">");
-
-				extraInfo.add(sb.toString());
-			}
-
-		} else if (fd.getType().equals(FieldDescriptor.Type.ENUM)) {
-			if (fd.getFullName().equals("eu.dnetlib.data.proto.OafEntity.type")) return;
-			metadata.add(asXmlElement(fd.getName(), ((EnumValueDescriptor) o).getName(), null, null));
-		} else {
-			metadata.add(asXmlElement(fd.getName(), o.toString(), null, null));
-		}
-	}
-
-	private StringBuilder dataInfoAsAttributes(final StringBuilder sb, final DataInfo dataInfo) {
-		sb.append(" inferred=\"" + dataInfo.getInferred() + "\"");
-		sb.append(" inferenceprovenance=\"" + dataInfo.getInferenceprovenance() + "\"");
-		sb.append(" provenanceaction=\"" + dataInfo.getProvenanceaction().getClassid() + "\"");
-		sb.append(" trust=\"" + dataInfo.getTrust() + "\" ");
-		return sb;
-	}
-
-	private List<String> buildContexts(final Type type) {
-		final List<String> res = Lists.newArrayList();
-
-		if ((contextMapper != null) && !contextMapper.isEmpty() && type.equals(Type.result)) {
-
-			XMLTag document = XMLDoc.newDocument(true).addRoot("contextRoot");
-
-			for (final String id : contextes) {
-
-				final StringTokenizer st = new StringTokenizer(id, "::");
-				String token = "";
-				while (st.hasMoreTokens()) {
-					token += st.nextToken();
-
-					final ContextDef def = contextMapper.get(token);
-
-					if (def == null) throw new IllegalStateException("cannot find context for id: " + token);
-
-					if (def.getName().equals("context")) {
-						final String xpath = "//context/@id='" + def.getId() + "'";
-						if (!document.gotoRoot().rawXpathBoolean(xpath, new Object())) {
-							document = addContextDef(document.gotoRoot(), def);
-						}
-					}
-
-					if (def.getName().equals("category")) {
-						final String rootId = StringUtils.substringBefore(def.getId(), "::");
-						document = addContextDef(document.gotoRoot().gotoTag("//context[./@id='" + rootId + "']", new Object()), def);
-					}
-
-					if (def.getName().equals("concept")) {
-						document = addContextDef(document, def).gotoParent();
-					}
-					token += "::";
-				}
-			}
-
-			for (final org.w3c.dom.Element x : document.gotoRoot().getChildElement()) {
-				try {
-					res.add(asStringElement(x));
-				} catch (final TransformerException e) {
-					throw new RuntimeException(e);
-				}
-			}
-		}
-
-		return res;
-	}
-
-	private XMLTag addContextDef(final XMLTag tag, final ContextDef def) {
-		tag.addTag(def.getName()).addAttribute("id", def.getId()).addAttribute("label", def.getLabel());
-		if ((def.getType() != null) && !def.getType().isEmpty()) {
-			tag.addAttribute("type", def.getType());
-		}
-		return tag;
-	}
-
-	private String asStringElement(final org.w3c.dom.Element element) throws TransformerException {
-		final StringWriter buffer = new StringWriter();
-		transformer.transform(new DOMSource(element), new StreamResult(buffer));
-		return buffer.toString();
-	}
-
-	@SuppressWarnings("unchecked")
-	private String getRelFundingTree(final String xmlTree) {
-		String funding = "<funding>";
-		try {
-			final Document ftree = new SAXReader().read(new StringReader(xmlTree));
-			funding = "<funding>";
-			// String _id = "";
-
-			funding += getFunderElement(ftree);
-
-			for (final Object o : Lists.reverse(ftree.selectNodes("//fundingtree//*[starts-with(local-name(),'funding_level_')]"))) {
-				final Element e = (Element) o;
-				final String _id = e.valueOf("./id");
-				funding += "<" + e.getName() + ">" + escapeXml(_id) + "</" + e.getName() + ">";
-				// _id += "::";
-			}
-		} catch (final DocumentException e) {
-			throw new IllegalArgumentException("unable to parse funding tree: " + xmlTree + "\n" + e.getMessage());
-		} finally {
-			funding += "</funding>";
-		}
-		return funding;
-	}
-
-	private String getFunderElement(final Document ftree) {
-		final String funderId = ftree.valueOf("//fundingtree/funder/id/text()");
-		final String funderShortName = ftree.valueOf("//fundingtree/funder/shortname/text()");
-		final String funderName = ftree.valueOf("//fundingtree/funder/name/text()");
-		final String funderJurisdiction = ftree.valueOf("//fundingtree/funder/jurisdiction/text()");
-
-		return "<funder id=\"" + escapeXml(funderId) + "\" shortname=\"" + escapeXml(funderShortName) + "\" name=\"" + escapeXml(funderName)
-				+ "\" jurisdiction=\"" + escapeXml(funderJurisdiction) + "\" />";
-	}
-
-	private void fillContextMap(final String xmlTree) {
-
-		Document fundingPath;
-		try {
-			fundingPath = new SAXReader().read(new StringReader(xmlTree));
-		} catch (final DocumentException e) {
-			throw new RuntimeException(e);
-		}
-		try {
-			final Node funder = fundingPath.selectSingleNode("//funder");
-			final String funderShortName = funder.valueOf("./shortname");
-			contextMapper.put(funderShortName, new ContextDef(funderShortName, funder.valueOf("./name"), "context", "funding"));
-			final Node level0 = fundingPath.selectSingleNode("//funding_level_0");
-			final String level0Id = Joiner.on("::").join(funderShortName, level0.valueOf("./name"));
-			contextMapper.put(level0Id, new ContextDef(level0Id, level0.valueOf("./description"), "category", ""));
-			final Node level1 = fundingPath.selectSingleNode("//funding_level_1");
-			if (level1 == null) {
-				contextes.add(level0Id);
-			} else {
-				final String level1Id = Joiner.on("::").join(level0Id, level1.valueOf("./name"));
-				contextMapper.put(level1Id, new ContextDef(level1Id, level1.valueOf("./description"), "concept", ""));
-				final Node level2 = fundingPath.selectSingleNode("//funding_level_2");
-				if (level2 == null) {
-					contextes.add(level1Id);
-				} else {
-					final String level2Id = Joiner.on("::").join(level1Id, level2.valueOf("./name"));
-					contextMapper.put(level2Id, new ContextDef(level2Id, level2.valueOf("./description"), "concept", ""));
-					contextes.add(level2Id);
-				}
-			}
-		} catch (final NullPointerException e) {
-			throw new IllegalArgumentException("malformed funding path: " + xmlTree, e);
-		}
-	}
-
-	private String asXmlElement(final String name, final String value, final Qualifier q, final DataInfo dataInfo) {
-		StringBuilder sb = new StringBuilder();
-		sb.append("<");
-		sb.append(name);
-		if (q != null) {
-			sb.append(getAttributes(q));
-		}
-		if (dataInfo != null) {
-			sb = dataInfoAsAttributes(sb, dataInfo);
-		}
-		if ((value == null) || value.isEmpty()) {
-			sb.append("/>");
-			return sb.toString();
-			// return "<" + name + getAttributes(q) + "/>";
-		}
-
-		sb.append(">");
-		// sb.append(escapeXml(Normalizer.normalize(value, Normalizer.Form.NFD)));
-		sb.append(escapeXml(value));
-		sb.append("</");
-		sb.append(name);
-		sb.append(">");
-
-		return sb.toString();
-		// return "<" + name + getAttributes(q) + ">" + escapeXml(value) + "</" + name + ">";
-	}
-
-	private String getAttributes(final Qualifier q) {
-		if (q == null) return "";
-
-		final StringBuilder sb = new StringBuilder();
-		for (final Entry<FieldDescriptor, Object> e : q.getAllFields().entrySet()) {
-			// sb.append(" " + e.getKey().getName() + "=\"" + escapeXml(e.getValue().toString()) + "\"");
-			sb.append(" ");
-			sb.append(e.getKey().getName());
-			sb.append("=\"");
-			sb.append(escapeXml(e.getValue().toString()));
-			sb.append("\"");
-		}
-		return sb.toString();
-	}
-
-	private Set<Entry<FieldDescriptor, Object>> filterFields(final GeneratedMessage fields, final Set<String> filter) {
-
-		if (filter != null) {
-			final Predicate<FieldDescriptor> p = new Predicate<FieldDescriptor>() {
-
-				@Override
-				public boolean apply(final FieldDescriptor descriptor) {
-					if (fields == null) return false;
-					final String name = descriptor.getName();
-					return filter.contains(name);
-				}
-			};
-			final Map<FieldDescriptor, Object> filtered = Maps.filterKeys(fields.getAllFields(), p);
-			// log.info(
-			// "filtered " + type.toString() + ": " + toString(filterEntries.keySet()) + "\n" +
-			// "builder  " + fields.getDescriptorForType().getFullName() + ": " + toString(fields.getAllFields().keySet()));
-			return filtered.entrySet();
-		}
-		return fields.getAllFields().entrySet();
-	}
-
-	public static String removePrefix(final String s) {
-		if (s.contains("|")) return StringUtils.substringAfter(s, "|");
-		return s;
-	}
-
-	public static String escapeXml(final String value) {
-		return value.replaceAll("&", "&amp;").replaceAll("<", "&lt;").replaceAll(">", "&gt;").replaceAll("\"", "&quot;").replaceAll("'", "&apos;");
-	}
-
-	private List<String> countersAsXml() {
-		final List<String> out = Lists.newArrayList();
-		for (final Entry<String, Integer> e : counters.entrySet()) {
-			out.add(String.format("<counter_%s value=\"%s\"/>", e.getKey(), e.getValue()));
-		}
-		return out;
-	}
-
-	private void incrementCounter(final String type) {
-		if (!counters.containsKey(type)) {
-			counters.put(type, 1);
-		} else {
-			counters.put(type, counters.get(type) + 1);
-		}
-	}
-
-	@Override
-	public String toString() {
-		final StringBuilder sb = new StringBuilder();
-		sb.append("################################################\n");
-		sb.append("ID: ").append(key).append("\n");
-		if (mainEntity != null) {
-			sb.append("MAIN ENTITY:\n").append(mainEntity.getEntity().toString() + "\n");
-		}
-		if (relations != null) {
-			sb.append("\nRELATIONS:\n");
-			for (final OafDecoder decoder : relations) {
-				sb.append(decoder.getOafRel().toString() + "\n");
-			}
-		}
-		if (children != null) {
-			sb.append("\nCHILDREN:\n");
-			for (final OafDecoder decoder : children) {
-				sb.append(decoder.getOafRel().toString() + "\n");
-			}
-		}
-		return sb.toString();
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/JobParams.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/JobParams.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/JobParams.java	(nonexistent)
@@ -1,69 +0,0 @@
-package eu.dnetlib.data.mapreduce;
-
-public class JobParams {
-
-	public static final int MAX_COUNTERS = 20;
-
-	public static final boolean WRITE_TO_WAL = false;
-
-	public static final String DEDUP_CONF = "dedup.conf";
-
-	public static final String MAPRED_JAR = "mapred.jar";
-	public static final String MAPRED_INPUT_DIR = "mapred.input.dir";
-	public static final String MAPRED_OUTPUT_DIR = "mapred.output.dir";
-	public static final String MAPRED_OUT_DIR = "mapred.out.dir";
-	public static final String MAPRED_WORKING_DIR = "mapred.working.dir";
-	public static final String MAPRED_LOCAL_DIR = "mapred.local.dir";
-
-	public static final String MONGO_INPUT_URI = "mongo.input.uri";
-	public static final String MONGO_OUTPUT_URI = "mongo.output.uri";
-	public static final String HBASE_TARGET_TABLE = "hbase.mapred.outputtable";
-	public static final String HBASE_TARGET_FAMILY = "hbase.target.family";
-	public static final String HBASE_TARGET_QUALIFIER = "hbase.target.qualifier";
-	public static final String HBASE_TARGET_TABLE_REGION_SPLITS = "hbase.target.table.region.splits";
-
-	public static final String HBASE_SOURCE_TABLE = "hbase.source.table";
-	public static final String HBASE_SOURCE_FAMILY = "hbase.source.family";
-	public static final String HBASE_SOURCE_QUALIFIER = "hbase.source.qualifier.body";
-
-	public static final String HBASE_DEST_TABLE = "hbase.dest.table";
-	public static final String HBASE_DEST_FAMILY = "hbase.dest.family";
-	public static final String HBASE_DEST_QUALIFIER = "hbase.dest.qualifier.body";
-
-	public static final String HBASE_IMPORT_XSLT = "hbase.import.xslt";
-
-	public static final String INDEX_SOLR_URL = "index.solr.url";
-	public static final String INDEX_SOLR_URL_ZK = "index.solr.url.zk";
-	public static final String INDEX_SOLR_COLLECTION = "index.solr.collection";
-	public static final String INDEX_SOLR_URL_LOCAL = "index.solr.url.local";
-	public static final String INDEX_SOLR_URL_LIST = "index.solr.url.list";
-	public static final String INDEX_SOLR_QUEUE_SIZE = "index.solr.queue.size";
-	public static final String INDEX_SOLR_THREAD_COUNT = "index.solr.thread.count";
-	public static final String INDEX_SEQ_FILE = "index.solr.seq.file";
-	public static final String INDEX_ROTTEN_FILE = "index.solr.rotten.records";
-	public static final String INDEX_LOCAL_FEEDING = "index.solr.local.feeding";
-	public static final String INDEX_FEED_SIMULATION_MODE = "index.solr.sim.mode";
-
-	public static final String INDEX_ADD_THRESHOLD = "index.add.threshold";
-	public static final String INDEX_XSLT = "index.xslt";
-	public static final String INDEX_REL_FILTERS = "index.rel.filters";
-	public static final String INDEX_ENTITY_LINKS = "index.entity.links";
-	public static final String INDEX_FEED_TYPE = "index.feed.type";
-	public static final String INDEX_FEED_MODE = "index.feed.mode";
-	public static final String INDEX_FEED_WF_MODE = "index.feed.wf.mode";
-	public static final String INDEX_FEED_TIME = "index.feed.timestamp";
-	public static final String INDEX_NUM_REDUCE_TASK = "index.num.reduce.task";
-	public static final String INDEX_DSID = "id";
-	public static final String INDEX_SHUTDOWN_WAIT = "index.shutdown.wait.time";
-	public static final String INDEX_BUFFER_FLUSH_TRESHOLD = "index.buffer.flush.threshold";
-
-	public static final String HDFS_TMP_DIR = "hdsf.tmp.directory";
-	public static final String DEDUP_SEQ_FILE = "dedup.seq.file";
-
-	public static final String STATS_ROTTEN_FILE = "stats.rotten.records";
-
-	public static final String OAI_FEED_DATE = "oai.feed.date";
-
-	public static final String DOCUMENT_DB_ROTTEN_FILE = "document.db.rotten.records";
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/EmptySolrDocumentException.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/EmptySolrDocumentException.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/EmptySolrDocumentException.java	(nonexistent)
@@ -1,7 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index;
-
-public class EmptySolrDocumentException extends Exception {
-
-	private static final long serialVersionUID = -7563164234424124258L;
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/PrepareFeedReducer.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/PrepareFeedReducer.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/PrepareFeedReducer.java	(nonexistent)
@@ -1,117 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index;
-
-import java.io.IOException;
-
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Reducer;
-import org.dom4j.DocumentException;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.hbase.index.config.ContextMapper;
-import eu.dnetlib.data.mapreduce.hbase.index.config.EntityConfigTable;
-import eu.dnetlib.data.mapreduce.hbase.index.config.IndexConfig;
-import eu.dnetlib.data.mapreduce.hbase.index.config.RelClasses;
-import eu.dnetlib.data.mapreduce.util.OafDecoder;
-import eu.dnetlib.data.mapreduce.util.OafHbaseUtils;
-import eu.dnetlib.data.mapreduce.util.OafRowKeyDecoder;
-import eu.dnetlib.data.mapreduce.util.XmlRecordFactory;
-
-public class PrepareFeedReducer extends Reducer<Text, ImmutableBytesWritable, Text, Text> {
-
-	private static final Log log = LogFactory.getLog(PrepareFeedReducer.class); // NOPMD by marko on 11/24/08 5:02 PM
-
-	private EntityConfigTable entityConfigTable;
-
-	private ContextMapper contextMapper = new ContextMapper();
-
-	private RelClasses relClasses;
-
-	private String schemaLocation;
-
-	private final boolean entityDefaults = true;
-	private final boolean relDefaults = false;
-	private final boolean childDefaults = false;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		final String json = context.getConfiguration().get(JobParams.INDEX_ENTITY_LINKS);
-		log.info(JobParams.INDEX_ENTITY_LINKS + ":\n" + json);
-		entityConfigTable = IndexConfig.load(json).getConfigMap();
-
-		schemaLocation = context.getConfiguration().get("oaf.schema.location");
-		log.info("schema location" + ":\n" + schemaLocation);
-
-		final String contextMap = context.getConfiguration().get("contextmap");
-		log.info("contextmap:\n" + contextMap);
-		try {
-			contextMapper = ContextMapper.fromXml(contextMap);
-		} catch (final DocumentException e) {
-			System.err.println("unable to parse contextMap: " + contextMap);
-		}
-
-		final String relClassJson = context.getConfiguration().get("relClasses");
-		log.info("relClassesJson:\n" + relClassJson);
-		relClasses = new RelClasses(relClassJson);
-		log.info("relClasses:\n" + relClasses);
-	}
-
-	@Override
-	protected void reduce(final Text key, final Iterable<ImmutableBytesWritable> values, final Context context) throws IOException, InterruptedException {
-		final OafRowKeyDecoder keyDecoder = OafRowKeyDecoder.decode(key.toString());
-
-		try {
-			final XmlRecordFactory builder =
-					new XmlRecordFactory(entityConfigTable, contextMapper, relClasses, schemaLocation, entityDefaults, relDefaults, childDefaults);
-
-			try {
-				decodeValues(context, keyDecoder, values, builder);
-
-				if (builder.isValid()) {
-					context.write(keyDecoder.getKeyAsText(), new Text(builder.build()));
-					// context.getCounter("entity", keyDecoder.getType().toString()).increment(1);
-				} else {
-					// System.err.println("invalid " + getType(key) + ": " + key.toString());
-					context.getCounter("missing body (reduce)", keyDecoder.getType().toString()).increment(1);
-				}
-			} catch (final OutOfMemoryError e) {
-				context.getCounter("error", e.getClass().getName()).increment(1);
-				log.error(String.format("memory error building entity\nid: '%s'\ncounters: %s", keyDecoder.getKey(), builder.getRelCounters()), e);
-				throw e;
-			}
-		} catch (final Exception e) {
-			context.getCounter("error", e.getClass().getName()).increment(1);
-			throw new RuntimeException(e);
-		}
-
-	}
-
-	private void decodeValues(final Context context,
-			final OafRowKeyDecoder keyDecoder,
-			final Iterable<ImmutableBytesWritable> values,
-			final XmlRecordFactory builder) {
-
-		for (final ImmutableBytesWritable bytes : values) {
-			final OafDecoder decoder = OafHbaseUtils.decode(bytes);
-
-			switch (decoder.getKind()) {
-			case entity:
-				builder.setMainEntity(decoder);
-				break;
-			case relation:
-
-				if (decoder.getOafRel().getChild()) {
-					builder.addChild(keyDecoder.getType(), decoder);
-				} else {
-					builder.addRelation(keyDecoder.getType(), decoder);
-				}
-				break;
-			default:
-				throw new IllegalArgumentException("unknow type: " + decoder.getKind());
-			}
-		}
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/IndexFeedMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/IndexFeedMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/IndexFeedMapper.java	(nonexistent)
@@ -1,229 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.Map.Entry;
-
-import org.apache.commons.codec.binary.Base64;
-import org.apache.commons.lang.exception.ExceptionUtils;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Mapper;
-import org.apache.solr.client.solrj.SolrServerException;
-import org.apache.solr.client.solrj.impl.CloudSolrServer;
-import org.apache.solr.client.solrj.response.SolrPingResponse;
-import org.apache.solr.client.solrj.response.UpdateResponse;
-import org.apache.solr.common.SolrInputDocument;
-
-import com.google.common.collect.Lists;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.functionality.index.solr.feed.InputDocumentFactory;
-import eu.dnetlib.functionality.index.solr.feed.StreamingInputDocumentFactory;
-import eu.dnetlib.miscutils.datetime.HumanTime;
-import eu.dnetlib.miscutils.functional.xml.ApplyXslt;
-
-public class IndexFeedMapper extends Mapper<Text, Text, Text, Text> {
-
-	private static final Log log = LogFactory.getLog(IndexFeedMapper.class); // NOPMD by marko on 11/24/08 5:02 PM
-
-	private InputDocumentFactory documentFactory;
-
-	private CloudSolrServer solrServer;
-
-	private String version;
-
-	private String dsId;
-
-	private int shutdownWaitTime = 10000;
-
-	private int bufferFlushThreshold = 100;
-
-	private ApplyXslt dmfToRecord;
-
-	private List<SolrInputDocument> buffer;
-
-	private int backoffTimeMs = 5000;
-
-	private boolean simulation = false;
-
-	private final static int MAX_INIT_RETRIES = 10;
-
-	private final static int MAX_FEED_RETRIES = 10;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-
-		logConfiguration(context.getConfiguration());
-
-		dsId = context.getConfiguration().get(JobParams.INDEX_DSID);
-		shutdownWaitTime = Integer.parseInt(context.getConfiguration().get(JobParams.INDEX_SHUTDOWN_WAIT));
-		bufferFlushThreshold = Integer.parseInt(context.getConfiguration().get(JobParams.INDEX_BUFFER_FLUSH_TRESHOLD));
-		documentFactory = new StreamingInputDocumentFactory();
-		version = InputDocumentFactory.getParsedDateField(context.getConfiguration().get(JobParams.INDEX_FEED_TIME));
-		buffer = Lists.newArrayList();
-		simulation = Boolean.parseBoolean(context.getConfiguration().get(JobParams.INDEX_FEED_SIMULATION_MODE));
-
-		final String xslt = new String(Base64.decodeBase64(context.getConfiguration().get(JobParams.INDEX_XSLT)));
-
-		log.info("got xslt: \n" + xslt);
-		log.info("got version: " + version);
-		log.info("simulation: " + simulation);
-		log.info("buffer size: " + bufferFlushThreshold);
-
-		dmfToRecord = new ApplyXslt(xslt);
-
-		final String baseURL = context.getConfiguration().get(JobParams.INDEX_SOLR_URL);
-		log.info("solr server baseURL: " + baseURL);
-
-		final String collection = context.getConfiguration().get(JobParams.INDEX_SOLR_COLLECTION);
-		log.info("solr server collection: " + collection);
-
-		int count = 0;
-		while (count <= MAX_INIT_RETRIES) {
-			try {
-				count++;
-				log.info("initializing solr server...");
-				solrServer = new CloudSolrServer(baseURL);
-
-				solrServer.connect();
-
-				solrServer.setParallelUpdates(true);
-				solrServer.setDefaultCollection(collection);
-
-				final SolrPingResponse rsp = solrServer.ping();
-
-				if (rsp.getStatus() != 0) throw new SolrServerException("bad init status: " + rsp.getStatus());
-				else {
-					break;
-				}
-
-			} catch (final Throwable e) {
-				if (solrServer != null) {
-					solrServer.shutdown();
-				}
-				context.getCounter("index init", e.getMessage()).increment(1);
-				log.error(String.format("failed to init solr client wait %dms, error:\n%s", backoffTimeMs, ExceptionUtils.getStackTrace(e)));
-
-				Thread.sleep(backoffTimeMs);
-			}
-		}
-
-		if (count >= MAX_INIT_RETRIES) throw new IOException("reached max retries trying to connect to solr server: " + MAX_INIT_RETRIES);
-	}
-
-	@Override
-	protected void map(final Text key, final Text value, final Context context) throws IOException, InterruptedException {
-
-		String indexRecord = "";
-		SolrInputDocument doc = null;
-
-		try {
-			indexRecord = dmfToRecord.evaluate(value.toString());
-			doc = documentFactory.parseDocument(version, indexRecord, dsId, "dnetResult");
-			if ((doc == null) || doc.isEmpty()) throw new EmptySolrDocumentException();
-
-		} catch (final Throwable e) {
-			context.getCounter("index feed", "skipped records").increment(1);
-			handleError(key, value, context, indexRecord, doc, e);
-			return;
-		}
-		int count = 0;
-		while (count <= MAX_FEED_RETRIES) {
-			count++;
-			try {
-				addDocument(context, doc);
-				return;
-			} catch (final Throwable e) {
-				context.getCounter("index feed", "retries").increment(1);
-				handleError(key, value, context, indexRecord, doc, e);
-				log.info(String.format("failed to feed documents, waiting %dms", backoffTimeMs));
-				Thread.sleep(backoffTimeMs);
-			}
-		}
-	}
-
-	private void addDocument(final Context context, final SolrInputDocument doc) throws SolrServerException, IOException, EmptySolrDocumentException {
-		buffer.add(doc);
-		if (buffer.size() >= bufferFlushThreshold) {
-			doAdd(buffer, context);
-		}
-	}
-
-	private void doAdd(final List<SolrInputDocument> buffer, final Context context) throws SolrServerException, IOException {
-		if (!simulation) {
-			final long start = System.currentTimeMillis();
-			final UpdateResponse rsp = solrServer.add(buffer);
-			final long stop = System.currentTimeMillis() - start;
-			log.info("feed time for " + buffer.size() + " records : " + HumanTime.exactly(stop) + "\n");
-
-			final int status = rsp.getStatus();
-
-			context.getCounter("index feed", "status code: " + status).increment(buffer.size());
-
-			if (status != 0) throw new SolrServerException("bad status: " + status);
-
-			for (final SolrInputDocument doc : buffer) {
-				context.getCounter("index entity", getEntityType(doc)).increment(1);
-			}
-		}
-		buffer.clear();
-	}
-
-	@Override
-	protected void cleanup(final Context context) throws IOException, InterruptedException {
-		super.cleanup(context);
-		try {
-			if (!buffer.isEmpty()) {
-				doAdd(buffer, context);
-			}
-			log.info("\nwaiting " + shutdownWaitTime + "ms before shutdown");
-			Thread.sleep(shutdownWaitTime);
-			solrServer.shutdown();
-		} catch (final SolrServerException e) {
-			log.error("couldn't shutdown server " + e.getMessage());
-		}
-	}
-
-	private void handleError(final Text key, final Text value, final Context context, final String indexRecord, final SolrInputDocument doc, final Throwable e)
-			throws IOException, InterruptedException {
-		context.getCounter("index feed", e.getClass().getName()).increment(1);
-		context.write(key, printRottenRecord(context.getTaskAttemptID().toString(), value, indexRecord, doc));
-		// e.printStackTrace(System.err);
-	}
-
-	private Text printRottenRecord(final String taskid, final Text value, final String indexRecord, final SolrInputDocument doc) {
-		return new Text("\n**********************************\n" + "task: " + taskid + "\n"
-				+ check("original", value.toString() + check("indexRecord", indexRecord) + check("solrDoc", doc)));
-	}
-
-	private String check(final String label, final Object value) {
-		if ((value != null) && !value.toString().isEmpty()) return "\n " + label + ":\n" + value + "\n";
-		return "\n";
-	}
-
-	private void logConfiguration(final Configuration conf) {
-		log.info("job configutation #################");
-		for (final Entry<String, String> e : conf) {
-			log.info("'" + e.getKey() + "' : '" + e.getValue() + "'");
-		}
-		log.info("end of job configutation #################\n\n");
-	}
-
-	private String getEntityType(final SolrInputDocument doc) {
-		if (!doc.containsKey("oaftype")) return "unknown";
-
-		final Type type = Type.valueOf(doc.getFieldValue("oaftype").toString());
-		switch (type) {
-		case result:
-			if (!doc.containsKey("resulttypeid")) return "result";
-			return doc.getFieldValue("resulttypeid").toString();
-		default:
-			return type.toString();
-		}
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/DedupIndexFeedMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/DedupIndexFeedMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/DedupIndexFeedMapper.java	(nonexistent)
@@ -1,239 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import org.apache.commons.collections.MapUtils;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-import org.apache.solr.client.solrj.SolrServerException;
-import org.apache.solr.client.solrj.impl.CloudSolrServer;
-import org.apache.solr.client.solrj.response.SolrPingResponse;
-import org.apache.solr.client.solrj.response.UpdateResponse;
-import org.apache.solr.common.SolrInputDocument;
-import org.dom4j.DocumentException;
-
-import com.google.common.collect.Lists;
-import com.googlecode.protobuf.format.JsonFormat;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.util.DedupUtils;
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.transform.SolrProtoMapper;
-import eu.dnetlib.functionality.index.solr.feed.InputDocumentFactory;
-import eu.dnetlib.miscutils.datetime.HumanTime;
-
-public class DedupIndexFeedMapper extends TableMapper<Text, Text> {
-
-	private static final Log log = LogFactory.getLog(DedupIndexFeedMapper.class); // NOPMD by marko on 11/24/08 5:02 PM
-
-	private CloudSolrServer solrServer;
-
-	private String dsId;
-
-	private String version;
-
-	private int shutdownWaitTime = 10000;
-
-	private int bufferFlushThreshold = 100;
-
-	private List<SolrInputDocument> buffer;
-
-	private int backoffTimeMs = 5000;
-
-	private boolean simulation = false;
-
-	private String entityType = null;
-
-	private String actionset = null;
-
-	private SolrProtoMapper mapper = null;
-
-	private final static int MAX_RETRIES = 10;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-
-		logConfiguration(context.getConfiguration());
-
-		shutdownWaitTime = Integer.parseInt(context.getConfiguration().get(JobParams.INDEX_SHUTDOWN_WAIT));
-		bufferFlushThreshold = Integer.parseInt(context.getConfiguration().get(JobParams.INDEX_BUFFER_FLUSH_TRESHOLD));
-		dsId = context.getConfiguration().get(JobParams.INDEX_DSID);
-		version = InputDocumentFactory.getParsedDateField(context.getConfiguration().get(JobParams.INDEX_FEED_TIME));
-		buffer = Lists.newArrayList();
-		simulation = Boolean.parseBoolean(context.getConfiguration().get(JobParams.INDEX_FEED_SIMULATION_MODE));
-		entityType = context.getConfiguration().get("entityType");
-		actionset = context.getConfiguration().get("actionset");
-
-		final String fields = context.getConfiguration().get("index.fields");
-
-		log.info("got fields: \n" + fields);
-		log.info("got dsId: " + dsId);
-		log.info("got version: " + version);
-		log.info("simulation: " + simulation);
-		log.info("entityType: " + entityType);
-		log.info("actionset: " + actionset);
-		log.info("buffer size: " + bufferFlushThreshold);
-
-		try {
-			mapper = new SolrProtoMapper(fields);
-		} catch (final DocumentException e) {
-			log.error("unable to parse fields: " + fields);
-			throw new IllegalArgumentException(e);
-		}
-
-		final String baseURL = context.getConfiguration().get(JobParams.INDEX_SOLR_URL);
-		log.info("solr server baseURL: " + baseURL);
-
-		final String collection = context.getConfiguration().get(JobParams.INDEX_SOLR_COLLECTION);
-		log.info("solr server collection: " + collection);
-
-		while (true) {
-			try {
-				log.info("initializing solr server...");
-				solrServer = new CloudSolrServer(baseURL);
-
-				solrServer.connect();
-
-				solrServer.setParallelUpdates(true);
-				solrServer.setDefaultCollection(collection);
-
-				final SolrPingResponse rsp = solrServer.ping();
-
-				if (rsp.getStatus() != 0) throw new SolrServerException("bad init status: " + rsp.getStatus());
-				else {
-					break;
-				}
-
-			} catch (final Throwable e) {
-				if (solrServer != null) {
-					solrServer.shutdown();
-				}
-				context.getCounter("index init", e.getMessage()).increment(1);
-				log.info(String.format("failed to init solr client wait %dms", backoffTimeMs));
-				Thread.sleep(backoffTimeMs);
-			}
-		}
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable key, final Result value, final Context context) throws IOException, InterruptedException {
-
-		SolrInputDocument doc = null;
-
-		final Map<byte[], byte[]> bMap = value.getFamilyMap(Bytes.toBytes(entityType));
-
-		if (MapUtils.isEmpty(bMap) || !bMap.containsKey(DedupUtils.BODY_B)) {
-			context.getCounter(entityType, "missing body");
-			return;
-		}
-
-		final Oaf oaf = Oaf.parseFrom(bMap.get(DedupUtils.BODY_B));
-
-		try {
-			doc = getDocument(oaf);
-		} catch (final Throwable e) {
-			handleError(key, JsonFormat.printToString(oaf), context, doc, e);
-			return;
-		}
-
-		int retries = 0;
-		while (retries < MAX_RETRIES) {
-			try {
-				addDocument(context, doc);
-				return;
-			} catch (final Throwable e) {
-				retries++;
-				context.getCounter("index feed", "retries").increment(1);
-				handleError(key, JsonFormat.printToString(oaf), context, doc, e);
-				log.info(String.format("failed to feed documents, waiting %dms", backoffTimeMs));
-				Thread.sleep(backoffTimeMs);
-			}
-		}
-		if (retries >= MAX_RETRIES)
-			throw new IOException("too many retries: " + retries);
-	}
-
-	private SolrInputDocument getDocument(final Oaf oaf) throws DocumentException {
-		final SolrInputDocument document = mapper.map(oaf, version, dsId, actionset);
-		document.addField("actionset", actionset);
-		return document;
-	}
-
-	private void addDocument(final Context context, final SolrInputDocument doc) throws SolrServerException, IOException {
-		if (!doc.isEmpty()) {
-
-			buffer.add(doc);
-			if (buffer.size() >= bufferFlushThreshold) {
-				doAdd(buffer, context);
-				// Thread.sleep(100);
-			}
-		} else {
-			context.getCounter("index feed", "skipped records").increment(1);
-		}
-	}
-
-	private void doAdd(final List<SolrInputDocument> buffer, final Context context) throws SolrServerException, IOException {
-		if (!simulation) {
-			final long start = System.currentTimeMillis();
-			final UpdateResponse rsp = solrServer.add(buffer);
-			final long stop = System.currentTimeMillis() - start;
-			log.info("feed time for " + buffer.size() + " records : " + HumanTime.exactly(stop) + "\n");
-
-			final int status = rsp.getStatus();
-			context.getCounter("index feed", "status code: " + status).increment(buffer.size());
-
-			if (status != 0) throw new SolrServerException("bad status: " + status);
-		}
-		buffer.clear();
-	}
-
-	@Override
-	protected void cleanup(final Context context) throws IOException, InterruptedException {
-		super.cleanup(context);
-		try {
-			if (!buffer.isEmpty()) {
-				doAdd(buffer, context);
-			}
-			log.info("\nwaiting " + shutdownWaitTime + "ms before shutdown");
-			Thread.sleep(shutdownWaitTime);
-			solrServer.shutdown();
-		} catch (final SolrServerException e) {
-			System.err.println("couldn't shutdown server " + e.getMessage());
-		}
-	}
-
-	private void handleError(final ImmutableBytesWritable key, final String value, final Context context, final SolrInputDocument doc, final Throwable e)
-			throws IOException, InterruptedException {
-		context.getCounter("index feed", e.getClass().getName()).increment(1);
-		context.write(new Text(key.copyBytes()), printRottenRecord(context.getTaskAttemptID().toString(), value, doc));
-		// e.printStackTrace(System.err);
-	}
-
-	private Text printRottenRecord(final String taskid, final String value, final SolrInputDocument doc) {
-		return new Text("\n**********************************\n" + "task: " + taskid + "\n"
-				+ check("original", value.toString() + check("solrDoc", doc)));
-	}
-
-	private String check(final String label, final Object value) {
-		if ((value != null) && !value.toString().isEmpty()) return "\n " + label + ":\n" + value + "\n";
-		return "\n";
-	}
-
-	private void logConfiguration(final Configuration conf) {
-		log.info("job configutation #################");
-		for (final Entry<String, String> e : conf) {
-			log.info("'" + e.getKey() + "' : '" + e.getValue() + "'");
-		}
-		log.info("end of job configutation #################\n\n");
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/DocumentDatabaseFeedJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/DocumentDatabaseFeedJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/DocumentDatabaseFeedJob.java	(nonexistent)
@@ -1,41 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index;
-
-import java.io.IOException;
-import java.util.Properties;
-
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
-import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
-import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.hbase.AbstractHBaseMapReduceJob;
-
-public class DocumentDatabaseFeedJob extends AbstractHBaseMapReduceJob {
-
-	@Override
-	public Job setJobDetails(final Job job, final Properties p) {
-
-		job.setInputFormatClass(SequenceFileInputFormat.class);
-		try {
-			FileInputFormat.setInputPaths(job, p.getProperty(JobParams.MAPRED_INPUT_DIR));
-			FileOutputFormat.setOutputPath(job, new Path(p.getProperty(JobParams.DOCUMENT_DB_ROTTEN_FILE)));
-		} catch (IOException e) {
-			throw new RuntimeException(e);
-		}
-		job.setMapperClass(DocumentDatabaseMapper.class);
-		job.setMapOutputKeyClass(Text.class);
-		job.setMapOutputValueClass(Text.class);
-
-		job.setNumReduceTasks(0);
-
-		job.getConfiguration().setBoolean("mapred.map.tasks.speculative.execution", false);
-		job.getConfiguration().setBoolean("mapreduce.map.speculative", false);
-
-		job.getConfiguration().setBoolean("mapred.compress.map.output", true);
-
-		return job;
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/ContextMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/ContextMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/ContextMapper.java	(nonexistent)
@@ -1,60 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index.config;
-
-import java.io.StringReader;
-import java.lang.reflect.Type;
-import java.util.HashMap;
-import java.util.List;
-
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.dom4j.Document;
-import org.dom4j.DocumentException;
-import org.dom4j.Node;
-import org.dom4j.io.SAXReader;
-
-import com.google.common.collect.Lists;
-import com.google.gson.Gson;
-import com.google.gson.reflect.TypeToken;
-
-public class ContextMapper extends HashMap<String, ContextDef> {
-
-	private static final long serialVersionUID = 2159682308502487305L;
-
-	private static final Log log = LogFactory.getLog(ContextMapper.class);
-
-	private final Type token = new TypeToken<List<ContextDef>>() {}.getType();
-
-	@SuppressWarnings("unchecked")
-	public void fromJson(final String json) {
-		super.clear();
-		if ((json != null) && !json.isEmpty()) {
-			for (ContextDef def : (List<ContextDef>) new Gson().fromJson(json, token)) {
-				super.put(def.getId(), def);
-			}
-		} else {
-			log.warn("unable to inizialize, empty source json string");
-		}
-	}
-
-	public static ContextMapper fromXml(final String xml) throws DocumentException {
-		final ContextMapper contextMapper = new ContextMapper();
-
-		Document doc = new SAXReader().read(new StringReader(xml));
-		for (Object o : doc.selectNodes("//entry")) {
-			Node node = (Node) o;
-			String id = node.valueOf("./@id");
-			String label = node.valueOf("./@label");
-			String name = node.valueOf("./@name");
-			String type = node.valueOf("./@type") + "";
-
-			contextMapper.put(id, new ContextDef(id, label, name, type));
-		}
-		return contextMapper;
-	}
-
-	@Override
-	public String toString() {
-		return new Gson().toJson(Lists.newArrayList(this.values()));
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/EntityConfigTable.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/EntityConfigTable.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/EntityConfigTable.java	(nonexistent)
@@ -1,32 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index.config;
-
-import java.util.Collection;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Set;
-
-import eu.dnetlib.data.mapreduce.util.RelDescriptor;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-
-public class EntityConfigTable extends HashMap<Type, EntityConfig> {
-
-	private static final long serialVersionUID = 6087987206844928698L;
-
-	public Collection<LinkDescriptor> getDescriptors(final Type source) {
-		return super.get(source).getLinks().values();
-	}
-
-	public LinkDescriptor getDescriptor(final Type type, final RelDescriptor relDescriptor) {
-		return super.get(type).getLinks().get(relDescriptor);
-	}
-
-	public Set<String> getFilter(final Type type, final RelDescriptor relDescriptor) {
-		final LinkDescriptor ld = getDescriptor(type, relDescriptor);
-		return ld != null ? ld.getFields() : new HashSet<String>();
-	}
-
-	public boolean includeDuplicates(final Type type) {
-		return super.get(type).getIndexDuplicates();
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/LinkDescriptor.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/LinkDescriptor.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/LinkDescriptor.java	(nonexistent)
@@ -1,87 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index.config;
-
-import java.util.Set;
-
-import com.google.gson.Gson;
-
-import eu.dnetlib.data.mapreduce.util.RelDescriptor;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-
-public class LinkDescriptor {
-
-	private RelDescriptor relDescriptor;
-	private Type target;
-	private boolean isChild = false;
-	private Set<String> fields;
-	private boolean symmetric = true;
-	private int max = -1;
-
-	public LinkDescriptor(final RelDescriptor relDescriptor, final Type target, final boolean isChild, final boolean symmetric, final Set<String> fields) {
-		super();
-		this.setRelDescriptor(relDescriptor);
-		this.setTarget(target);
-		this.setChild(isChild);
-		this.setFields(fields);
-		this.setSymmetric(symmetric);
-	}
-
-	public LinkDescriptor(final RelDescriptor relDescriptor, final Type target) {
-		super();
-		this.setRelDescriptor(relDescriptor);
-		this.setTarget(target);
-	}
-
-	public Type getTarget() {
-		return target;
-	}
-
-	public boolean isChild() {
-		return isChild;
-	}
-
-	public void setTarget(final Type target) {
-		this.target = target;
-	}
-
-	public void setChild(final boolean isChild) {
-		this.isChild = isChild;
-	}
-
-	public Set<String> getFields() {
-		return fields;
-	}
-
-	public void setFields(final Set<String> fields) {
-		this.fields = fields;
-	}
-
-	public boolean isSymmetric() {
-		return symmetric;
-	}
-
-	public void setSymmetric(final boolean symmetric) {
-		this.symmetric = symmetric;
-	}
-
-	@Override
-	public String toString() {
-		return new Gson().toJson(this);
-	}
-
-	public RelDescriptor getRelDescriptor() {
-		return relDescriptor;
-	}
-
-	public void setRelDescriptor(final RelDescriptor relDescriptor) {
-		this.relDescriptor = relDescriptor;
-	}
-
-	public int getMax() {
-		return max;
-	}
-
-	public void setMax(final int max) {
-		this.max = max;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/IndexConfig.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/IndexConfig.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/IndexConfig.java	(nonexistent)
@@ -1,74 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index.config;
-
-import java.io.StringReader;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-import com.google.common.base.Function;
-import com.google.common.collect.Maps;
-import com.google.common.collect.Sets;
-import com.google.gson.Gson;
-import com.typesafe.config.Config;
-import com.typesafe.config.ConfigFactory;
-import com.typesafe.config.ConfigObject;
-import com.typesafe.config.ConfigValue;
-
-import eu.dnetlib.data.mapreduce.OptionalConfig;
-import eu.dnetlib.data.mapreduce.util.RelDescriptor;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-
-public class IndexConfig extends OptionalConfig {
-
-	public static IndexConfig load(final String s) {
-		return new IndexConfig(ConfigFactory.parseReader(new StringReader(s)));
-	}
-
-	public IndexConfig(final Config config) {
-		super(config);
-	}
-
-	@SuppressWarnings("unchecked")
-	public EntityConfigTable getConfigMap() {
-		final ConfigObject modelMap = getConfig().getObject("index.conf");
-		final Map<String, EntityConfig> map = Maps.transformValues(modelMap, new Function<ConfigValue, EntityConfig>() {
-
-			@Override
-			public EntityConfig apply(final ConfigValue value) {
-
-				final HashMap<RelDescriptor, LinkDescriptor> links = Maps.newHashMap();
-				for (final Object o : (List<?>) ((Map<String, ?>) value.unwrapped()).get("links")) {
-					final Map<String, ?> map = (Map<String, ?>) o;
-
-					final RelDescriptor relDescriptor = new RelDescriptor((String) map.get("relType"));
-					final Set<String> fields = Sets.newHashSet((List<String>) map.get("fields"));
-					final Type target = Type.valueOf((String) map.get("targetEntity"));
-					final Boolean isChild = ((String) map.get("expandAs")).equals("child");
-					final Boolean symmetric = (Boolean) map.get("symmetric");
-
-					final LinkDescriptor ld = new LinkDescriptor(relDescriptor, target, isChild, symmetric, fields);
-
-					if (map.containsKey("max")) {
-						ld.setMax((Integer) map.get("max"));
-					}
-
-					links.put(relDescriptor, ld);
-				}
-				return new EntityConfig((Boolean) ((Map<String, ?>) value.unwrapped()).get("dups"), links);
-			}
-		});
-
-		final EntityConfigTable res = new EntityConfigTable();
-		for (final String key : map.keySet()) {
-			res.put(Type.valueOf(key), map.get(key));
-		}
-		return res;
-	}
-
-	@Override
-	public String toString() {
-		return new Gson().toJson(this);
-	}
-
-}

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/IndexConfig.java
___________________________________________________________________
Deleted: svn:mime-type
## -1 +0,0 ##
-text/plain
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/RelClasses.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/RelClasses.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/RelClasses.java	(nonexistent)
@@ -1,39 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index.config;
-
-import java.lang.reflect.Type;
-import java.util.List;
-import java.util.Map;
-
-import com.google.common.collect.Maps;
-import com.google.gson.Gson;
-import com.google.gson.GsonBuilder;
-import com.google.gson.reflect.TypeToken;
-
-public class RelClasses {
-
-	private final Map<String, String> map = Maps.newHashMap();
-
-	private final Type typeToken = new TypeToken<List<String>>() {}.getType();
-
-	public static RelClasses fromJSon(String json) {
-		return new RelClasses(json);
-	}
-
-	public RelClasses(String json) {
-		List<String> list = new Gson().fromJson(json, typeToken);
-		for (String s : list) {
-			String[] split = s.split(":");
-			map.put(split[0], split[1]);
-		}
-	}
-
-	public String getInverse(String relClass) {
-		return map.get(relClass);
-	}
-
-	@Override
-	public String toString() {
-		return new GsonBuilder().setPrettyPrinting().create().toJson(map);
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/EntityConfig.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/EntityConfig.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/EntityConfig.java	(nonexistent)
@@ -1,35 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index.config;
-
-import java.util.HashMap;
-
-import eu.dnetlib.data.mapreduce.util.RelDescriptor;
-
-public class EntityConfig {
-
-	private Boolean includeDuplicates;
-
-	private HashMap<RelDescriptor, LinkDescriptor> links;
-
-	public EntityConfig(final Boolean includeDuplicates, final HashMap<RelDescriptor, LinkDescriptor> links) {
-		super();
-		this.includeDuplicates = includeDuplicates;
-		this.links = links;
-	}
-
-	public Boolean getIndexDuplicates() {
-		return includeDuplicates;
-	}
-
-	public void setIndexDuplicates(final Boolean indexDuplicates) {
-		this.includeDuplicates = indexDuplicates;
-	}
-
-	public HashMap<RelDescriptor, LinkDescriptor> getLinks() {
-		return links;
-	}
-
-	public void setLinks(final HashMap<RelDescriptor, LinkDescriptor> links) {
-		this.links = links;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/EntityLinkTable.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/EntityLinkTable.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/EntityLinkTable.java	(nonexistent)
@@ -1,28 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index.config;
-
-import java.util.Collection;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Set;
-
-import eu.dnetlib.data.proto.RelTypeProtos.RelType;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-
-@Deprecated
-public class EntityLinkTable extends HashMap<Type, HashMap<RelType, LinkDescriptor>> {
-
-	private static final long serialVersionUID = -5045143790736657223L;
-
-	public Collection<LinkDescriptor> getDescriptors(final Type source) {
-		return super.get(source).values();
-	}
-
-	public LinkDescriptor getDescriptor(final Type type, final RelType relType) {
-		return super.get(type).get(relType);
-	}
-
-	public Set<String> getFilter(final Type type, final RelType relType) {
-		final LinkDescriptor ld = getDescriptor(type, relType);
-		return ld != null ? ld.getFields() : new HashSet<String>();
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/ContextDef.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/ContextDef.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/config/ContextDef.java	(nonexistent)
@@ -1,49 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index.config;
-
-public class ContextDef {
-
-	private String id;
-	private String label;
-	private String name;
-	private String type;
-
-	public ContextDef(final String id, final String label, final String name, final String type) {
-		super();
-		this.setId(id);
-		this.setLabel(label);
-		this.setName(name);
-		this.setType(type);
-	}
-
-	public String getLabel() {
-		return label;
-	}
-
-	public void setLabel(final String label) {
-		this.label = label;
-	}
-
-	public String getId() {
-		return id;
-	}
-
-	public void setId(final String id) {
-		this.id = id;
-	}
-
-	public String getName() {
-		return name;
-	}
-
-	public void setName(final String name) {
-		this.name = name;
-	}
-
-	public String getType() {
-		return type;
-	}
-
-	public void setType(final String type) {
-		this.type = type;
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/PrepareFeedMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/PrepareFeedMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/PrepareFeedMapper.java	(nonexistent)
@@ -1,211 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index;
-
-import java.io.IOException;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Counter;
-
-import com.google.protobuf.InvalidProtocolBufferException;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.hbase.index.config.EntityConfigTable;
-import eu.dnetlib.data.mapreduce.hbase.index.config.IndexConfig;
-import eu.dnetlib.data.mapreduce.hbase.index.config.LinkDescriptor;
-import eu.dnetlib.data.mapreduce.hbase.index.config.RelClasses;
-import eu.dnetlib.data.mapreduce.util.OafRowKeyDecoder;
-import eu.dnetlib.data.mapreduce.util.RelDescriptor;
-import eu.dnetlib.data.mapreduce.util.UpdateMerger;
-import eu.dnetlib.data.proto.KindProtos.Kind;
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.proto.OafProtos.OafEntity;
-import eu.dnetlib.data.proto.OafProtos.OafRel;
-import eu.dnetlib.data.proto.OafProtos.OafRelOrBuilder;
-import eu.dnetlib.data.proto.RelTypeProtos.SubRelType;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-
-public class PrepareFeedMapper extends TableMapper<Text, ImmutableBytesWritable> {
-
-	private EntityConfigTable entityConfigTable;
-
-	private RelClasses relClasses;
-
-	private Text outKey;
-
-	private ImmutableBytesWritable ibw;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		final String json = context.getConfiguration().get(JobParams.INDEX_ENTITY_LINKS);
-		System.out.println(JobParams.INDEX_ENTITY_LINKS + ":\n" + json);
-		entityConfigTable = IndexConfig.load(json).getConfigMap();
-
-		final String contextMap = context.getConfiguration().get("contextmap");
-		System.out.println("contextmap:\n" + contextMap);
-
-		final String relClassJson = context.getConfiguration().get("relClasses");
-		System.out.println("relClassesJson:\n" + relClassJson);
-		relClasses = RelClasses.fromJSon(relClassJson);
-		System.out.println("relClasses:\n" + relClasses);
-
-		outKey = new Text();
-		ibw = new ImmutableBytesWritable();
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable keyIn, final Result value, final Context context) throws IOException, InterruptedException {
-
-		final OafRowKeyDecoder keyDecoder = OafRowKeyDecoder.decode(keyIn.copyBytes());
-
-		final Type type = keyDecoder.getType();
-		final Oaf oaf = mergeUpdates(value, context, type, keyDecoder);
-
-		if (isValid(oaf)) {
-
-			if (!deletedByInference(oaf) || entityConfigTable.includeDuplicates(type)) {
-				emit(new String(keyIn.copyBytes()), context, oaf);
-
-				incrementCounter(context, Kind.entity.toString(), getEntityType(oaf, type), 1);
-
-				for (final LinkDescriptor ld : entityConfigTable.getDescriptors(type)) {
-
-					final Map<byte[], byte[]> columnMap = value.getFamilyMap(Bytes.toBytes(ld.getRelDescriptor().getIt()));
-
-					if (hasData(columnMap)) {
-						emitRelationship(oaf.getEntity(), context, columnMap, ld);
-						incrementCounter(context, type.toString(), ld.getRelDescriptor().getIt(), columnMap.size());
-					} // else {
-					// incrementCounter(context, type.toString(), ld.getRelDescriptor().getIt() + "_empty", 1);
-					// }
-				}
-			} else {
-				incrementCounter(context, "deleted by inference", type.toString(), 1);
-			}
-		} else {
-			incrementCounter(context, "missing body (map)", type.toString(), 1);
-		}
-	}
-
-	private Oaf mergeUpdates(final Result value, final Context context, final Type type, final OafRowKeyDecoder keyDecoder)
-			throws InvalidProtocolBufferException {
-		try {
-			return UpdateMerger.mergeBodyUpdates(context, value.getFamilyMap(Bytes.toBytes(type.toString())));
-		} catch (final InvalidProtocolBufferException e) {
-			System.err.println(String.format("Unable to parse proto (Type: %s) in row: %s", type.toString(), keyDecoder.getKey()));
-			throw e;
-		}
-	}
-
-	private void emitRelationship(final OafEntity cachedTarget, final Context context, final Map<byte[], byte[]> columnMap, final LinkDescriptor ld)
-			throws IOException, InterruptedException {
-
-		final Oaf.Builder oafBuilder = Oaf.newBuilder().setKind(Kind.relation);
-
-		// iterates the column map
-		for (final Entry<byte[], byte[]> e : columnMap.entrySet()) {
-
-			final Oaf oaf = decodeProto(context, e.getValue());
-			if (!isValid(oaf)) {
-				incrementCounter(context, "invalid oaf rel", ld.getRelDescriptor().getIt(), 1);
-			} else if (!deletedByInference(oaf)) {
-
-				final OafRel.Builder relBuilder = OafRel.newBuilder(oaf.getRel());
-
-				if (ld.isSymmetric()) {
-					final RelDescriptor rd = ld.getRelDescriptor();
-					relBuilder.setCachedTarget(cachedTarget).setRelType(rd.getRelType()).setSubRelType(rd.getSubRelType());
-				}
-
-				if (ld.getRelDescriptor().getIt().contains(SubRelType.dedup.toString()) && isDedupSelf(relBuilder)) {
-					incrementCounter(context, "avoid to emit dedup self", ld.getRelDescriptor().getIt(), 1);
-					continue;
-				}
-
-				final OafRel oafRel = relBuilder.setChild(ld.isChild()).build();
-
-				// String rowKey = patchTargetId(target, OafRelDecoder.decode(oafRel).getRelTargetId());
-
-				emit(ld.isSymmetric() ? oafRel.getTarget() : oafRel.getSource(), context, merge(oafBuilder, oaf).setRel(oafRel).build());
-			} else {
-				incrementCounter(context, "deleted by inference", ld.getRelDescriptor().getIt(), 1);
-			}
-		}
-	}
-
-	private String patchTargetId(final Type target, final String id) {
-		return id.replaceFirst("^.*\\|", target.getNumber() + "|");
-	}
-
-	private Oaf.Builder merge(final Oaf.Builder builder, final Oaf prototype) {
-		return builder.setDataInfo(prototype.getDataInfo()).setTimestamp(prototype.getTimestamp());
-	}
-
-	private boolean isDedupSelf(final OafRelOrBuilder rel) {
-		return rel.getSource().contains(rel.getTarget());
-	}
-
-	private boolean hasData(final Map<byte[], byte[]> columnMap) {
-		return (columnMap != null) && !columnMap.isEmpty();
-	}
-
-	private boolean isValid(final Oaf oaf) {
-		return (oaf != null) && oaf.isInitialized();
-	}
-
-	private boolean deletedByInference(final Oaf oaf) {
-		return oaf.getDataInfo().getDeletedbyinference();
-	}
-
-	private Oaf decodeProto(final Context context, final byte[] body) {
-		try {
-			return Oaf.parseFrom(body);
-		} catch (final InvalidProtocolBufferException e) {
-			e.printStackTrace(System.err);
-			context.getCounter("decodeProto", e.getClass().getName()).increment(1);
-		}
-		return null;
-	}
-
-	// private byte[] prefix(final Type name, final byte[] bytes) {
-	// return concat(concat(name.toString().getBytes(), PrepareFeedJob.bSEPARATOR), bytes);
-	// }
-
-	private void emit(final String key, final Context context, final Oaf oaf) throws IOException, InterruptedException {
-		// Text keyOut = new Text(Hashing.murmur3_128().hashString(key).toString());
-		outKey.set(key);
-		ibw.set(oaf.toByteArray());
-
-		context.write(outKey, ibw);
-	}
-
-	// protected byte[] concat(final byte[] a, final byte[] b) {
-	// byte[] c = new byte[a.length + b.length];
-	// System.arraycopy(a, 0, c, 0, a.length);
-	// System.arraycopy(b, 0, c, a.length, b.length);
-	//
-	// return c;
-	// }
-
-	private void incrementCounter(final Context context, final String k, final String t, final int n) {
-		getCounter(context, k, t).increment(n);
-	}
-
-	private Counter getCounter(final Context context, final String k, final String t) {
-		return context.getCounter(k, t);
-	}
-
-	private String getEntityType(final Oaf oaf, final Type type) {
-		switch (type) {
-		case result:
-			return oaf.getEntity().getResult().getMetadata().getResulttype().getClassid();
-		default:
-			return type.toString();
-		}
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/DocumentDatabaseMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/DocumentDatabaseMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/index/DocumentDatabaseMapper.java	(nonexistent)
@@ -1,60 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.index;
-
-import java.io.IOException;
-import java.io.StringReader;
-import java.util.Map.Entry;
-
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Mapper;
-import org.dom4j.Document;
-import org.dom4j.io.SAXReader;
-
-public class DocumentDatabaseMapper extends Mapper<Text, Text, Text, Text> {
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		logConfiguration(context.getConfiguration());
-	}
-
-	@Override
-	protected void map(final Text key, final Text value, final Context context) throws IOException, InterruptedException {
-		try {
-			final SAXReader reader = new SAXReader();
-
-			final Document doc = reader.read(new StringReader(value.toString()));
-
-			final String id = doc.valueOf("//dri:objIdentifier");
-
-			addDocument(context, id, value.toString());
-
-			context.getCounter("Document Database", "records").increment(1);
-		} catch (Exception e) {
-			context.getCounter("Document Database", "errors").increment(1);
-			context.write(key, value);
-			e.printStackTrace(System.err);
-		}
-
-	}
-
-	private void addDocument(final Context context, final String id, final String string) {
-		// TODO
-
-		// ADD the document to the DB
-
-	}
-
-	@Override
-	protected void cleanup(final Context context) throws IOException, InterruptedException {
-		super.cleanup(context);
-	}
-
-	private void logConfiguration(final Configuration conf) {
-		System.out.println("job configutation #################");
-		for (Entry<String, String> e : conf) {
-			System.out.println("'" + e.getKey() + "' : '" + e.getValue() + "'");
-		}
-		System.out.println("end of job configutation #################\n\n");
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/AbstractHBaseMapReduceJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/AbstractHBaseMapReduceJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/AbstractHBaseMapReduceJob.java	(nonexistent)
@@ -1,78 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase;
-
-import java.io.IOException;
-import java.util.Map.Entry;
-import java.util.Properties;
-
-import javax.annotation.Resource;
-
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.conf.Configured;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.mapreduce.Job;
-import org.springframework.beans.factory.BeanNameAware;
-
-import eu.dnetlib.data.hadoop.config.ClusterName;
-import eu.dnetlib.data.hadoop.config.ConfigurationEnumerator;
-import eu.dnetlib.data.mapreduce.HadoopJob;
-
-public abstract class AbstractHBaseMapReduceJob extends Configured implements HadoopJob, BeanNameAware {
-
-	protected static final Log log = LogFactory.getLog(AbstractHBaseMapReduceJob.class);
-
-	protected String jobName;
-
-	@Resource
-	protected ConfigurationEnumerator configurationEnumerator;
-
-	protected abstract Job setJobDetails(Job job, Properties p) throws Exception;
-
-	@Override
-	public Job setJobDetails(ClusterName name, Properties p) {
-		try {
-			final Job job = createJobCommon(name, p);
-			return setJobDetails(job, p);
-		} catch (Exception e) {
-			throw new RuntimeException("unable to define Job: " + getClass().getSimpleName(), e);
-		}
-	}
-
-	protected Job createJobCommon(ClusterName name, Properties p) throws IOException {
-
-		Job job = new Job(configurationEnumerator.get(name), getClass().getSimpleName());
-
-		merge(job, p);
-
-		return job;
-	}
-
-	protected void merge(final Job job, final Properties p) {
-		for (Entry<Object, Object> e : p.entrySet()) {
-			job.getConfiguration().set((String) e.getKey(), (String) e.getValue());
-		}
-	}
-
-	protected void deleteHdfsFile(Job job, Path path) {
-		try {
-			FileSystem hdfs = FileSystem.get(job.getConfiguration());
-			if (hdfs.exists(path)) {
-				hdfs.delete(path, true);
-			}
-		} catch (IOException e) {
-			e.printStackTrace();
-		}
-	}
-
-	@Override
-	public void setBeanName(String name) {
-		this.jobName = name;
-	}
-
-	@Override
-	public String getName() {
-		return jobName;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfiguration.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfiguration.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfiguration.java	(nonexistent)
@@ -1,129 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai.config;
-
-import java.util.Collection;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-import com.google.common.base.Joiner;
-import com.google.common.base.Predicate;
-import com.google.common.collect.Collections2;
-import com.google.common.collect.Sets;
-import com.google.gson.Gson;
-import com.google.gson.GsonBuilder;
-
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.MDFInfo;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.PublisherField;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.SetInfo;
-
-/**
- * An instance of this class represents an OAI Configuration profile.
- * 
- * @author alessia
- * 
- */
-public class OAIConfiguration {
-
-	/**
-	 * Map for OAI sets. Keys: setSpec, values: SetInfo instances.
-	 */
-	private Map<String, SetInfo> setsMap;
-
-	/**
-	 * Map for metadata formats. Keys: metadata prefix, values: MDFInfo instances.
-	 */
-	private Map<String, MDFInfo> mdFormatsMap;
-
-	private List<PublisherField> fields;
-
-	private List<String> fieldNames;
-
-	/**
-	 * Used to generate the OAI identifiers compliant to the protocol. See
-	 * http://www.openarchives.org/OAI/openarchivesprotocol.html#UniqueIdentifier. See property services.oai.publisher.id.scheme.
-	 */
-	private String idScheme;
-	/**
-	 * Used to generate the OAI identifiers compliant to the protocol. See
-	 * http://www.openarchives.org/OAI/openarchivesprotocol.html#UniqueIdentifier. See property services.oai.publisher.id.namespace.
-	 */
-	private String idNamespace;
-
-	private Set<MDFInfo> sourcesMDF = Sets.newHashSet();
-
-	public Collection<PublisherField> getFieldsFor(final String format, final String layout, final String interpretation) {
-		final String mdRef = Joiner.on("-").join(format, layout, interpretation);
-		return Collections2.filter(this.getFields(), new Predicate<PublisherField>() {
-
-			@Override
-			public boolean apply(final PublisherField theField) {
-				if (theField.getSources() == null) return false;
-				return theField.getSources().containsKey(mdRef);
-			}
-
-		});
-	}
-
-	public Map<String, SetInfo> getSetsMap() {
-		return setsMap;
-	}
-
-	public void setSetsMap(final Map<String, SetInfo> setsMap) {
-		this.setsMap = setsMap;
-	}
-
-	public Map<String, MDFInfo> getMdFormatsMap() {
-		return mdFormatsMap;
-	}
-
-	public void setMdFormatsMap(final Map<String, MDFInfo> mdFormatsMap) {
-		this.mdFormatsMap = mdFormatsMap;
-	}
-
-	public List<PublisherField> getFields() {
-		return fields;
-	}
-
-	public void setFields(final List<PublisherField> fields) {
-		this.fields = fields;
-	}
-
-	public List<String> getFieldNames() {
-		return fieldNames;
-	}
-
-	public void setFieldNames(final List<String> fieldNames) {
-		this.fieldNames = fieldNames;
-	}
-
-	public Set<MDFInfo> getSourcesMDF() {
-		return sourcesMDF;
-	}
-
-	public void setSourcesMDF(final Set<MDFInfo> sourcesMDF) {
-		this.sourcesMDF = sourcesMDF;
-	}
-
-	public String getIdScheme() {
-		return idScheme;
-	}
-
-	public void setIdScheme(final String idScheme) {
-		this.idScheme = idScheme;
-	}
-
-	public String getIdNamespace() {
-		return idNamespace;
-	}
-
-	public void setIdNamespace(final String idNamespace) {
-		this.idNamespace = idNamespace;
-	}
-
-	@Override
-	public String toString() {
-		Gson gson = new GsonBuilder().setPrettyPrinting().create();
-		return gson.toJson(this);
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfigurationReader.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfigurationReader.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfigurationReader.java	(nonexistent)
@@ -1,47 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai.config;
-
-import java.util.List;
-
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.MDFInfo;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.PublisherField;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.SetInfo;
-
-public interface OAIConfigurationReader {
-
-	/**
-	 * Constant names of fields used by the publisher to store records in the OAIStores. If you want an index over them, you need to specify
-	 * it in the OAI Publisher Configuration profile.
-	 */
-	public final static String ID_FIELD = "objIdentifier";
-	public final static String DATESTAMP_FIELD = "datestamp";
-	public final static String SET_FIELD = "set";
-	public final static String DELETED_FIELD = "deleted";
-	public final static String BODY_FIELD = "body";
-	public final static String UPDATED_FIELD = "updated";
-	public final static String LAST_COLLECTION_DATE_FIELD = "lastCollectionDate";
-
-	public List<SetInfo> getSets();
-
-	public List<SetInfo> getSets(final boolean onlyEnabled);
-
-	public List<String> getSetSpecs();
-
-	public List<String> getSetSpecs(final boolean onlyEnabled);
-
-	public SetInfo getSetInfo(final String setSpec);
-
-	public List<MDFInfo> getMetadataFormatInfo();
-
-	public List<MDFInfo> getMetadataFormatInfo(final boolean onlyEnabled);
-
-	public MDFInfo getMetadataFormatInfo(final String mdPrefix);
-
-	public List<String> getFieldNames();
-
-	public List<PublisherField> getFields();
-
-	public List<PublisherField> getFields(final String format, final String interpretation, final String layout);
-
-	public List<MDFInfo> getSourceMetadataFormats();
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfigurationStringReader.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfigurationStringReader.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfigurationStringReader.java	(nonexistent)
@@ -1,182 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai.config;
-
-import java.io.IOException;
-import java.util.List;
-
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-
-import com.google.common.base.Function;
-import com.google.common.base.Predicate;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.MDFInfo;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.OaiPublisherRuntimeException;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.PublisherField;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.SetInfo;
-
-/**
- * Instances of this class reads the OAI configuration from a string, which is the configuration profile passed in as a string.
- * 
- * @author alessia
- * 
- */
-public class OAIConfigurationStringReader implements OAIConfigurationReader {
-
-	private static final Log log = LogFactory.getLog(OAIConfigurationStringReader.class); // NOPMD by marko on 11/24/08 5:02 PM
-
-	private OAIConfiguration oaiConfiguration;
-
-	private OAIConfigurationParser configurationParser = new OAIConfigurationParser();
-
-	public OAIConfigurationStringReader() {
-		oaiConfiguration = null;
-	}
-
-	public OAIConfigurationStringReader(final String profile) {
-		this.readConfiguration(profile);
-	}
-
-	public void readConfiguration(final String profile) {
-		log.info("Reading OAI configuration profile");
-		log.debug(profile);
-		try {
-			oaiConfiguration = configurationParser.getConfiguration(profile);
-		} catch (IOException e) {
-			throw new OaiPublisherRuntimeException("Could not read OAI configuration profile", e);
-		}
-	}
-
-	public boolean isConfigurationLoaded() {
-		return this.oaiConfiguration != null;
-	}
-
-	@Override
-	public List<SetInfo> getSets() {
-		if (isConfigurationLoaded()) return Lists.newArrayList(this.oaiConfiguration.getSetsMap().values());
-		else throw new OaiPublisherRuntimeException("Configuration is not loaded");
-	}
-
-	@Override
-	public List<SetInfo> getSets(final boolean onlyEnabled) {
-		if (isConfigurationLoaded()) return Lists.newArrayList(Iterables.filter(this.oaiConfiguration.getSetsMap().values(), new Predicate<SetInfo>() {
-
-			/**
-			 * {@inheritDoc}
-			 * 
-			 * @see com.google.common.base.Predicate#apply(java.lang.Object)
-			 */
-			@Override
-			public boolean apply(final SetInfo setInfo) {
-				if (onlyEnabled) return setInfo.isEnabled();
-				else return true;
-			}
-
-		}));
-		else throw new OaiPublisherRuntimeException("Configuration is not loaded");
-	}
-
-	@Override
-	public List<String> getSetSpecs() {
-		if (isConfigurationLoaded()) return Lists.newArrayList(this.oaiConfiguration.getSetsMap().keySet());
-		else throw new OaiPublisherRuntimeException("Configuration is not loaded");
-	}
-
-	@Override
-	public List<String> getSetSpecs(final boolean onlyEnabled) {
-		if (isConfigurationLoaded()) {
-			if (!onlyEnabled) return this.getSetSpecs();
-			List<SetInfo> enabled = this.getSets(true);
-			return Lists.newArrayList(Iterables.transform(enabled, new Function<SetInfo, String>() {
-
-				@Override
-				public String apply(final SetInfo setInfo) {
-					return setInfo.getSetSpec();
-				}
-
-			}));
-		} else throw new OaiPublisherRuntimeException("Configuration is not loaded");
-	}
-
-	@Override
-	public List<MDFInfo> getSourceMetadataFormats() {
-		if (isConfigurationLoaded()) return Lists.newArrayList(this.oaiConfiguration.getSourcesMDF());
-		else throw new OaiPublisherRuntimeException("Configuration is not loaded");
-	}
-
-	@Override
-	public SetInfo getSetInfo(final String setSpec) {
-		if (isConfigurationLoaded()) return this.oaiConfiguration.getSetsMap().get(setSpec);
-		else throw new OaiPublisherRuntimeException("Configuration is not loaded");
-	}
-
-	@Override
-	public List<MDFInfo> getMetadataFormatInfo() {
-		if (isConfigurationLoaded()) return Lists.newArrayList(this.oaiConfiguration.getMdFormatsMap().values());
-		else throw new OaiPublisherRuntimeException("Configuration is not loaded");
-	}
-
-	@Override
-	public List<MDFInfo> getMetadataFormatInfo(final boolean onlyEnabled) {
-		if (isConfigurationLoaded()) {
-			if (!onlyEnabled) return this.getMetadataFormatInfo();
-			else return Lists.newArrayList(Iterables.filter(this.oaiConfiguration.getMdFormatsMap().values(), new Predicate<MDFInfo>() {
-
-				/**
-				 * {@inheritDoc}
-				 * 
-				 * @see com.google.common.base.Predicate#apply(java.lang.Object)
-				 */
-				@Override
-				public boolean apply(final MDFInfo mdfInfo) {
-					if (onlyEnabled) return mdfInfo.isEnabled();
-					else return true;
-				}
-
-			}));
-
-		} else throw new OaiPublisherRuntimeException("Configuration is not loaded");
-	}
-
-	@Override
-	public MDFInfo getMetadataFormatInfo(final String mdPrefix) {
-		if (isConfigurationLoaded()) return this.oaiConfiguration.getMdFormatsMap().get(mdPrefix);
-		else throw new OaiPublisherRuntimeException("Configuration is not loaded");
-	}
-
-	@Override
-	public List<PublisherField> getFields() {
-		if (isConfigurationLoaded()) return this.oaiConfiguration.getFields();
-		else throw new OaiPublisherRuntimeException("Configuration is not loaded");
-	}
-
-	@Override
-	public List<PublisherField> getFields(final String format, final String interpretation, final String layout) {
-		if (isConfigurationLoaded()) return Lists.newArrayList(this.oaiConfiguration.getFieldsFor(format, layout, interpretation));
-		else throw new OaiPublisherRuntimeException("Configuration is not loaded");
-	}
-
-	@Override
-	public List<String> getFieldNames() {
-		if (isConfigurationLoaded()) return this.oaiConfiguration.getFieldNames();
-		else throw new OaiPublisherRuntimeException("Configuration is not loaded");
-	}
-
-	public OAIConfiguration getOaiConfiguration() {
-		return oaiConfiguration;
-	}
-
-	public void setOaiConfiguration(final OAIConfiguration oaiConfiguration) {
-		this.oaiConfiguration = oaiConfiguration;
-	}
-
-	public OAIConfigurationParser getConfigurationParser() {
-		return configurationParser;
-	}
-
-	public void setConfigurationParser(final OAIConfigurationParser configurationParser) {
-		this.configurationParser = configurationParser;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfigurationParser.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfigurationParser.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/config/OAIConfigurationParser.java	(nonexistent)
@@ -1,237 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai.config;
-
-import java.io.IOException;
-import java.io.StringReader;
-import java.util.List;
-import java.util.Map;
-
-import javax.xml.stream.XMLInputFactory;
-import javax.xml.stream.XMLStreamConstants;
-import javax.xml.stream.XMLStreamException;
-import javax.xml.stream.XMLStreamReader;
-import javax.xml.transform.stream.StreamSource;
-
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-
-import com.google.common.collect.ArrayListMultimap;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-import com.google.common.collect.Multimap;
-
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.MDFInfo;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.OaiPublisherRuntimeException;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.PublisherField;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.SetInfo;
-
-/**
- * Parses an XML document representing the OAI configuration profile and creates the corresponding OAIConfiguration object.
- * 
- * @author alessia
- * 
- */
-public class OAIConfigurationParser {
-
-	private static final Log log = LogFactory.getLog(OAIConfigurationParser.class); // NOPMD by marko on 11/24/08 5:02 PM
-
-	private final ThreadLocal<XMLInputFactory> factory = new ThreadLocal<XMLInputFactory>() {
-
-		@Override
-		protected XMLInputFactory initialValue() {
-			return XMLInputFactory.newInstance();
-		}
-	};
-
-	public OAIConfiguration getConfiguration(final String configurationProfile) throws IOException {
-		log.info("Parsing OAI configuration profile");
-		log.debug(configurationProfile);
-		OAIConfiguration config = new OAIConfiguration();
-		Map<String, SetInfo> setsMap = Maps.newHashMap();
-		Map<String, MDFInfo> mdFormatsMap = Maps.newHashMap();
-		List<String> indexNames = Lists.newArrayList();
-		List<PublisherField> fields = Lists.newArrayList();
-		try {
-			final XMLStreamReader parser = factory.get().createXMLStreamReader(new StreamSource(new StringReader(configurationProfile)));
-			log.debug("Configuration profile read by " + parser.getClass().getCanonicalName());
-
-			while (parser.hasNext()) {
-				int event = parser.next();
-				if (event == XMLStreamConstants.START_ELEMENT) {
-					final String localName = parser.getLocalName();
-					if (localName.equals("IDSCHEME")) {
-						config.setIdScheme(parser.getElementText());
-					} else if (localName.equals("IDNAMESPACE")) {
-						config.setIdNamespace(parser.getElementText());
-					} else if (localName.equals("OAISET")) {
-						boolean inSet = true;
-						SetInfo setInfo = new SetInfo();
-						String enabled = parser.getAttributeValue(null, "enabled");
-						setInfo.setEnabled(Boolean.parseBoolean(enabled));
-						while (parser.hasNext() && inSet) {
-							event = parser.next();
-							if (event == XMLStreamConstants.START_ELEMENT) {
-								String setElementName = parser.getLocalName();
-								String setElementValue = parser.getElementText();
-								this.handleSetInfo(setInfo, setElementName, setElementValue);
-							}
-							if ((event == XMLStreamConstants.END_ELEMENT) && parser.getLocalName().equals("OAISET")) {
-								inSet = false;
-								setsMap.put(setInfo.getSetSpec(), setInfo);
-							}
-						}
-					} else {
-						if (localName.equals("METADATAFORMAT")) {
-							boolean inMetadata = true;
-							MDFInfo mdfInfo = new MDFInfo();
-							String exportable = parser.getAttributeValue(null, "exportable");
-							mdfInfo.setEnabled(Boolean.parseBoolean(exportable));
-							String mdPrefix = parser.getAttributeValue(null, "metadataPrefix");
-							mdfInfo.setPrefix(mdPrefix);
-							while (parser.hasNext() && inMetadata) {
-								event = parser.next();
-								if (event == XMLStreamConstants.START_ELEMENT) {
-									String mdfElementName = parser.getLocalName();
-									if (mdfElementName.equals("SOURCE_METADATA_FORMAT")) {
-										this.handleSourceMDF(mdfInfo, parser);
-										config.getSourcesMDF().add(mdfInfo);
-									} else {
-										String mdfElementValue = parser.getElementText();
-										this.handleMDFInfo(mdfInfo, mdfElementName, mdfElementValue);
-									}
-								}
-								if ((event == XMLStreamConstants.END_ELEMENT) && parser.getLocalName().equals("METADATAFORMAT")) {
-									inMetadata = false;
-									mdFormatsMap.put(mdPrefix, mdfInfo);
-								}
-							}
-						} else {
-							// INDICES
-							if (localName.equals("INDEX")) {
-								boolean inIndex = true;
-								PublisherField publisherField = new PublisherField();
-								String indexName = parser.getAttributeValue(null, "name");
-								String repeatable = parser.getAttributeValue(null, "repeatable");
-								boolean isRepeatable = Boolean.valueOf(repeatable);
-								indexNames.add(indexName);
-								publisherField.setFieldName(indexName);
-								publisherField.setRepeatable(isRepeatable);
-								// now let's set the SOURCES
-								Multimap<String, String> fieldSources = ArrayListMultimap.create();
-								while (parser.hasNext() && inIndex) {
-									event = parser.next();
-									if (event == XMLStreamConstants.START_ELEMENT) {
-										String currentElementName = parser.getLocalName();
-										this.handleIndex(fieldSources, indexName, parser, currentElementName);
-									}
-									if ((event == XMLStreamConstants.END_ELEMENT) && parser.getLocalName().equals("INDEX")) {
-										inIndex = false;
-									}
-								}
-								publisherField.setSources(fieldSources);
-								fields.add(publisherField);
-
-							}
-						}
-					}
-				}
-			}
-			config.setFields(fields);
-			config.setFieldNames(indexNames);
-			config.setMdFormatsMap(mdFormatsMap);
-			config.setSetsMap(setsMap);
-			return config;
-		} catch (final XMLStreamException e) {
-			throw new OaiPublisherRuntimeException(e);
-		}
-
-	}
-
-	/**
-	 * Sets information about the indices.
-	 * 
-	 * @param fieldSources
-	 * @param indexName
-	 * @param parser
-	 * @param currentLocalName
-	 */
-	private void handleIndex(final Multimap<String, String> fieldSources, final String indexName, final XMLStreamReader parser, final String currentLocalName) {
-		if (currentLocalName.equals("SOURCE")) {
-			MDFInfo indexSource = new MDFInfo();
-			this.handleSourceMDF(indexSource, parser);
-			String key = indexSource.getSourceFormatName() + "-" + indexSource.getSourceFormatLayout() + "-" + indexSource.getSourceFormatInterpretation();
-			fieldSources.put(key, parser.getAttributeValue(null, "path"));
-		} else {
-			log.warn("I do not know how to handle INDEX element with name " + currentLocalName);
-		}
-	}
-
-	/**
-	 * Sets information about the source metadata format in the given instance of MDFInfo.
-	 * 
-	 * @param mdfInfo
-	 * @param parser
-	 */
-	private void handleSourceMDF(final MDFInfo mdfInfo, final XMLStreamReader parser) {
-		String interpretation = parser.getAttributeValue(null, "interpretation");
-		String layout = parser.getAttributeValue(null, "layout");
-		String name = parser.getAttributeValue(null, "name");
-		mdfInfo.setSourceFormatInterpretation(interpretation);
-		mdfInfo.setSourceFormatLayout(layout);
-		mdfInfo.setSourceFormatName(name);
-	}
-
-	/**
-	 * Sets information in the MDFInfo instance based on the element name.
-	 * 
-	 * @param mdfInfo
-	 * @param mdfElementName
-	 * @param mdfElementValue
-	 */
-	private void handleMDFInfo(final MDFInfo mdfInfo, final String elementName, final String elementValue) {
-		if (elementName.equals("NAMESPACE")) {
-			mdfInfo.setNamespace(elementValue);
-		} else {
-			if (elementName.equals("SCHEMA")) {
-				mdfInfo.setSchema(elementValue);
-			} else {
-				if (elementName.equals("TRANSFORMATION_RULE")) {
-					mdfInfo.setTransformationRuleID(elementValue);
-				} else {
-					if (elementName.equals("BASE_QUERY")) {
-						mdfInfo.setBaseQuery(elementValue);
-					} else {
-						log.warn("I do not know how to handle Metadata Format element with name " + elementName + " and value " + elementValue);
-					}
-				}
-			}
-		}
-
-	}
-
-	/**
-	 * Sets information in the SetInfo instance based on the element name.
-	 * 
-	 * @param setInfo
-	 * @param elementName
-	 * @param elementValue
-	 */
-	private void handleSetInfo(final SetInfo setInfo, final String elementName, final String elementValue) {
-		if (elementName.equals("spec")) {
-			setInfo.setSetSpec(elementValue);
-		} else {
-			if (elementName.equals("name")) {
-				setInfo.setSetName(elementValue);
-			} else {
-				if (elementName.equals("description")) {
-					setInfo.setSetDescription(elementValue);
-				} else {
-					if (elementName.equals("query")) {
-						setInfo.setQuery(elementValue);
-					} else {
-						log.warn("I do not know how to handle Set element with name " + elementName + " and value " + elementValue);
-					}
-				}
-			}
-		}
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/OAIFeedJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/OAIFeedJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/OAIFeedJob.java	(nonexistent)
@@ -1,42 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai;
-
-import java.io.IOException;
-import java.util.Properties;
-
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.io.NullWritable;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
-import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
-import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.hbase.AbstractHBaseMapReduceJob;
-
-public class OAIFeedJob extends AbstractHBaseMapReduceJob {
-
-	@Override
-	public Job setJobDetails(final Job job, final Properties p) {
-
-		job.setInputFormatClass(SequenceFileInputFormat.class);
-		try {
-			FileInputFormat.setInputPaths(job, p.getProperty(JobParams.MAPRED_INPUT_DIR));
-			FileOutputFormat.setOutputPath(job, new Path(p.getProperty(JobParams.MAPRED_OUTPUT_DIR)));
-		} catch (IOException e) {
-			throw new RuntimeException(e);
-		}
-		job.setMapperClass(OaiFeedMapper.class);
-		job.setMapOutputKeyClass(NullWritable.class);
-		job.setMapOutputValueClass(NullWritable.class);
-
-		job.setNumReduceTasks(0);
-
-		job.getConfiguration().setBoolean("mapred.map.tasks.speculative.execution", false);
-		job.getConfiguration().setBoolean("mapreduce.map.speculative", false);
-
-		job.getConfiguration().setBoolean("mapred.compress.map.output", true);
-
-		return job;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/OaiPublisherRuntimeException.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/OaiPublisherRuntimeException.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/OaiPublisherRuntimeException.java	(nonexistent)
@@ -1,23 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai.utils;
-
-public class OaiPublisherRuntimeException extends RuntimeException {
-
-	private static final long serialVersionUID = -4295850381530160166L;
-
-	public OaiPublisherRuntimeException() {
-		super();
-	}
-
-	public OaiPublisherRuntimeException(final Throwable e) {
-		super(e);
-	}
-
-	public OaiPublisherRuntimeException(final String msg, final Throwable e) {
-		super(msg, e);
-	}
-
-	public OaiPublisherRuntimeException(final String msg) {
-		super(msg);
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/RecordFieldsExtractor.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/RecordFieldsExtractor.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/RecordFieldsExtractor.java	(nonexistent)
@@ -1,131 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai.utils;
-
-import java.io.StringReader;
-import java.util.List;
-import java.util.Map.Entry;
-
-import org.dom4j.Document;
-import org.dom4j.DocumentException;
-import org.dom4j.Node;
-import org.dom4j.io.SAXReader;
-
-import com.google.common.base.Function;
-import com.google.common.collect.ArrayListMultimap;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Multimap;
-
-/**
- * An instance of this class can parse an XML record and extract the information needed to store the record in a publisher store.
- * 
- * 
- * @author alessia
- * 
- */
-public class RecordFieldsExtractor {
-
-	/**
-	 * List of the indices of the target store.
-	 */
-	private List<PublisherField> storeIndices;
-
-	private String duplicateXPath;
-	private boolean skipDuplicates = false;
-
-	/**
-	 * Parses the record and returns a map where a key is the name of an index, the value is the value in the record at the xpath specificed
-	 * in this.storeIndices.
-	 * 
-	 * @param record
-	 *            the XML string to parse.
-	 * @return a Multimap describing the values to be indexed for this record.
-	 */
-	@SuppressWarnings({ "unchecked", "rawtypes" })
-	public Multimap<String, String> extractFields(final String record) {
-		Multimap<String, String> recordProps = ArrayListMultimap.create();
-		try {
-			Document doc = new SAXReader().read(new StringReader(record));
-			if (skipDuplicates && isDuplicate(doc)) {
-				recordProps.put("duplicate", "true");
-			}
-			for (PublisherField field : this.storeIndices) {
-				for (Entry<String, String> indexEntry : field.getSources().entries()) {
-					// each xpath can return a list of nodes or strings, depending on the xpath
-					String xpath = indexEntry.getValue();
-
-					List xPathResult = doc.selectNodes(xpath);
-					if ((xPathResult != null) && !xPathResult.isEmpty()) {
-						if (containsStrings(xPathResult)) {
-							recordProps.putAll(field.getFieldName(), xPathResult);
-						} else {
-							if (containsNodes(xPathResult)) {
-								recordProps.putAll(field.getFieldName(), Iterables.transform(xPathResult, new Function<Object, String>() {
-
-									@Override
-									public String apply(final Object obj) {
-										if (obj == null) return "";
-										Node node = (Node) obj;
-										return node.getText();
-									}
-								}));
-							}
-						}
-					}
-				}
-			}
-
-		} catch (DocumentException e) {
-			recordProps = null;
-		}
-		return recordProps;
-	}
-
-	public boolean isDuplicate(final Document doc) {
-		return doc.selectSingleNode(duplicateXPath) != null;
-	}
-
-	@SuppressWarnings("rawtypes")
-	private boolean containsStrings(final List objects) {
-		Object first = objects.get(0);
-		return first instanceof String;
-	}
-
-	@SuppressWarnings("rawtypes")
-	private boolean containsNodes(final List objects) {
-		Object first = objects.get(0);
-		return first instanceof Node;
-	}
-
-	public List<PublisherField> getStoreIndices() {
-		return storeIndices;
-	}
-
-	public void setStoreIndices(final List<PublisherField> storeIndices) {
-		this.storeIndices = storeIndices;
-	}
-
-	public RecordFieldsExtractor(final List<PublisherField> storeIndices) {
-		super();
-		this.storeIndices = storeIndices;
-	}
-
-	public RecordFieldsExtractor() {
-		super();
-	}
-
-	public String getDuplicateXPath() {
-		return duplicateXPath;
-	}
-
-	public void setDuplicateXPath(final String duplicateXPath) {
-		this.duplicateXPath = duplicateXPath;
-	}
-
-	public boolean isSkipDuplicates() {
-		return skipDuplicates;
-	}
-
-	public void setSkipDuplicates(final boolean skipDuplicates) {
-		this.skipDuplicates = skipDuplicates;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/PublisherField.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/PublisherField.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/PublisherField.java	(nonexistent)
@@ -1,81 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai.utils;
-
-import java.util.Map.Entry;
-
-import com.google.common.collect.Multimap;
-
-/**
- * Instances of this class represent fields to be explicitely managed by the OAI Publisher, for example for indexing purposes.
- * 
- * @author alessia
- * 
- */
-public class PublisherField {
-
-	/**
-	 * Name of the field.
-	 */
-	private String fieldName;
-	/**
-	 * True if the value of the field is a list (i.e. repeatable values). False otherwise.
-	 **/
-	private boolean repeatable;
-	/**
-	 * Information about where to take values for this field.
-	 * <p>
-	 * Keys are metadata format: name-layout-interpretation; values are xpaths.
-	 * </p>
-	 */
-	private Multimap<String, String> sources;
-
-	public String getFieldName() {
-		return fieldName;
-	}
-
-	public void setFieldName(final String fieldName) {
-		this.fieldName = fieldName;
-	}
-
-	public boolean isRepeatable() {
-		return repeatable;
-	}
-
-	public void setRepeatable(final boolean repeatable) {
-		this.repeatable = repeatable;
-	}
-
-	public Multimap<String, String> getSources() {
-		return sources;
-	}
-
-	public void setSources(final Multimap<String, String> sources) {
-		this.sources = sources;
-	}
-
-	public PublisherField() {
-		super();
-		// TODO Auto-generated constructor stub
-	}
-
-	public PublisherField(final String fieldName, final boolean repeatable, final Multimap<String, String> sources) {
-		super();
-		this.fieldName = fieldName;
-		this.repeatable = repeatable;
-		this.sources = sources;
-	}
-
-	@Override
-	public String toString() {
-		StringBuilder sb = new StringBuilder();
-		sb.append("fieldname: " + getFieldName() + "\n");
-		sb.append("repeatible: " + isRepeatable() + "\n");
-		sb.append("sources { ");
-		for (Entry<String, String> e : getSources().entries()) {
-			sb.append("\n\t" + e.getKey() + ":" + e.getValue() + "\n");
-		}
-		sb.append("}");
-		return sb.toString();
-		// Gson gson = new GsonBuilder().setPrettyPrinting().create();
-		// return gson.toJson(this);
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/MDFInfo.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/MDFInfo.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/MDFInfo.java	(nonexistent)
@@ -1,236 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai.utils;
-
-public class MDFInfo {
-
-	/**
-	 * prefix, schema and namespace are properties describing the metadata format exported via OAI PMH.
-	 */
-	private String prefix;
-	private String schema;
-	private String namespace;
-
-	/**
-	 * sourceFormat* are properties describing the metadata format to use as input to deliver records in the <prefix> format.
-	 * sourceFormatName may be the same as prefix, for example in cases when we have a oai_dc store and we want to deliver oai_dc records.
-	 */
-	private String sourceFormatName;
-	private String sourceFormatLayout;
-	private String sourceFormatInterpretation;
-
-	/**
-	 * query to filter out records to be delivered.
-	 */
-	private String baseQuery;
-	/**
-	 * identificator of the TDS profile containing an XSLT to be applied to records of <sourceFormatName> before delivering. The
-	 * transformation transforms them into the <prefix> metadata format. This property can be null/empty in cases when the delivered format
-	 * is the same of the source format.
-	 */
-	private String transformationRuleID;
-
-	/**
-	 * Tells if the metadata format is enabled for export. If not, it won't appear in the ListMetadataFormats response.
-	 */
-	private boolean enabled;
-
-	public MDFInfo() {}
-
-	public MDFInfo(final String prefix, final String schema, final String namespace, final String sourceFormatName, final String sourceFormatLayout,
-			final String sourceFormatInterpretation, final String baseQuery, final boolean enabled) {
-		super();
-		initCommonFields(prefix, schema, namespace, sourceFormatName, sourceFormatLayout, sourceFormatInterpretation, baseQuery);
-		this.enabled = enabled;
-	}
-
-	/**
-	 * This constructor is the same as the above, but in this case "enabled" is a string, so we can use the SplittedQueryExecutor to
-	 * generate instances of this class from the information stored in the D-Net IS.
-	 * 
-	 * @param prefix
-	 * @param schema
-	 * @param namespace
-	 * @param sourceFormatName
-	 * @param sourceFormatLayout
-	 * @param sourceFormatInterpretation
-	 * @param baseQuery
-	 * @param enabled
-	 */
-	public MDFInfo(final String prefix, final String schema, final String namespace, final String sourceFormatName, final String sourceFormatLayout,
-			final String sourceFormatInterpretation, final String baseQuery, final String enabled) {
-		super();
-		initCommonFields(prefix, schema, namespace, sourceFormatName, sourceFormatLayout, sourceFormatInterpretation, baseQuery);
-		this.enabled = Boolean.valueOf(enabled);
-	}
-
-	public MDFInfo(final String prefix, final String schema, final String namespace, final String sourceFormatName, final String sourceFormatLayout,
-			final String sourceFormatInterpretation, final String baseQuery, final String tdsID, final boolean enabled) {
-		this(prefix, schema, namespace, sourceFormatName, sourceFormatLayout, sourceFormatInterpretation, baseQuery, enabled);
-		transformationRuleID = tdsID;
-	}
-
-	/**
-	 * This constructor is the same as the above, but in this case "enabled" is a string, so we can use the SplittedQueryExecutor to
-	 * generate instances of this class from the information stored in the D-Net IS.
-	 * 
-	 * @param prefix
-	 * @param schema
-	 * @param namespace
-	 * @param sourceFormatName
-	 * @param sourceFormatLayout
-	 * @param sourceFormatInterpretation
-	 * @param baseQuery
-	 * @param tdsID
-	 * @param enabled
-	 */
-	public MDFInfo(final String prefix, final String schema, final String namespace, final String sourceFormatName, final String sourceFormatLayout,
-			final String sourceFormatInterpretation, final String baseQuery, final String tdsID, final String enabled) {
-		this(prefix, schema, namespace, sourceFormatName, sourceFormatLayout, sourceFormatInterpretation, baseQuery, enabled);
-		transformationRuleID = tdsID;
-	}
-
-	private void initCommonFields(final String prefix,
-			final String schema,
-			final String namespace,
-			final String sourceFormatName,
-			final String sourceFormatLayout,
-			final String sourceFormatInterpretation,
-			final String baseQuery) {
-		this.prefix = prefix;
-		this.schema = schema;
-		this.namespace = namespace;
-		this.sourceFormatName = sourceFormatName;
-		this.sourceFormatLayout = sourceFormatLayout;
-		this.sourceFormatInterpretation = sourceFormatInterpretation;
-		this.baseQuery = baseQuery;
-	}
-
-	@Override
-	public String toString() {
-		return "MDFInfo [prefix=" + prefix + ", schema=" + schema + ", namespace=" + namespace + ", sourceFormatName=" + sourceFormatName
-				+ ", sourceFormatLayout=" + sourceFormatLayout + ", sourceFormatInterpretation=" + sourceFormatInterpretation + ", baseQuery=" + baseQuery
-				+ ", transformationRuleID=" + transformationRuleID + ", enabled=" + enabled + "]";
-	}
-
-	public String getPrefix() {
-		return prefix;
-	}
-
-	public void setPrefix(final String prefix) {
-		this.prefix = prefix;
-	}
-
-	public String getSchema() {
-		return schema;
-	}
-
-	public void setSchema(final String schema) {
-		this.schema = schema;
-	}
-
-	public String getNamespace() {
-		return namespace;
-	}
-
-	public void setNamespace(final String namespace) {
-		this.namespace = namespace;
-	}
-
-	public String getSourceFormatName() {
-		return sourceFormatName;
-	}
-
-	public void setSourceFormatName(final String sourceFormatName) {
-		this.sourceFormatName = sourceFormatName;
-	}
-
-	public String getSourceFormatLayout() {
-		return sourceFormatLayout;
-	}
-
-	public void setSourceFormatLayout(final String sourceFormatLayout) {
-		this.sourceFormatLayout = sourceFormatLayout;
-	}
-
-	public String getSourceFormatInterpretation() {
-		return sourceFormatInterpretation;
-	}
-
-	public void setSourceFormatInterpretation(final String sourceFormatInterpretation) {
-		this.sourceFormatInterpretation = sourceFormatInterpretation;
-	}
-
-	public String getBaseQuery() {
-		return baseQuery;
-	}
-
-	public void setBaseQuery(final String baseQuery) {
-		this.baseQuery = baseQuery;
-	}
-
-	public String getTransformationRuleID() {
-		return transformationRuleID;
-	}
-
-	public void setTransformationRuleID(final String transformationRuleID) {
-		this.transformationRuleID = transformationRuleID;
-	}
-
-	public boolean isEnabled() {
-		return enabled;
-	}
-
-	public void setEnabled(final boolean enabled) {
-		this.enabled = enabled;
-	}
-
-	@Override
-	public int hashCode() {
-		final int prime = 31;
-		int result = 1;
-		result = (prime * result) + ((baseQuery == null) ? 0 : baseQuery.hashCode());
-		result = (prime * result) + (enabled ? 1231 : 1237);
-		result = (prime * result) + ((namespace == null) ? 0 : namespace.hashCode());
-		result = (prime * result) + ((prefix == null) ? 0 : prefix.hashCode());
-		result = (prime * result) + ((schema == null) ? 0 : schema.hashCode());
-		result = (prime * result) + ((sourceFormatInterpretation == null) ? 0 : sourceFormatInterpretation.hashCode());
-		result = (prime * result) + ((sourceFormatLayout == null) ? 0 : sourceFormatLayout.hashCode());
-		result = (prime * result) + ((sourceFormatName == null) ? 0 : sourceFormatName.hashCode());
-		result = (prime * result) + ((transformationRuleID == null) ? 0 : transformationRuleID.hashCode());
-		return result;
-	}
-
-	@Override
-	public boolean equals(final Object obj) {
-		if (this == obj) return true;
-		if (obj == null) return false;
-		if (!(obj instanceof MDFInfo)) return false;
-		MDFInfo other = (MDFInfo) obj;
-		if (baseQuery == null) {
-			if (other.baseQuery != null) return false;
-		} else if (!baseQuery.equals(other.baseQuery)) return false;
-		if (enabled != other.enabled) return false;
-		if (namespace == null) {
-			if (other.namespace != null) return false;
-		} else if (!namespace.equals(other.namespace)) return false;
-		if (prefix == null) {
-			if (other.prefix != null) return false;
-		} else if (!prefix.equals(other.prefix)) return false;
-		if (schema == null) {
-			if (other.schema != null) return false;
-		} else if (!schema.equals(other.schema)) return false;
-		if (sourceFormatInterpretation == null) {
-			if (other.sourceFormatInterpretation != null) return false;
-		} else if (!sourceFormatInterpretation.equals(other.sourceFormatInterpretation)) return false;
-		if (sourceFormatLayout == null) {
-			if (other.sourceFormatLayout != null) return false;
-		} else if (!sourceFormatLayout.equals(other.sourceFormatLayout)) return false;
-		if (sourceFormatName == null) {
-			if (other.sourceFormatName != null) return false;
-		} else if (!sourceFormatName.equals(other.sourceFormatName)) return false;
-		if (transformationRuleID == null) {
-			if (other.transformationRuleID != null) return false;
-		} else if (!transformationRuleID.equals(other.transformationRuleID)) return false;
-		return true;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/SetCollection.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/SetCollection.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/SetCollection.java	(nonexistent)
@@ -1,48 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai.utils;
-
-import java.util.List;
-
-/**
- * Collection of sets.
- * 
- * @author michele
- * 
- */
-public interface SetCollection {
-
-	/**
-	 * Get all sets.
-	 * 
-	 * @return list of SetInfo instances
-	 */
-	List<SetInfo> getAllSets();
-
-	/**
-	 * Test if a set is in this collection.
-	 * 
-	 * @param set
-	 *            set spec
-	 * @return true if the set is in this collection
-	 */
-	boolean containSet(final String set);
-
-	/**
-	 * Obtains the set query.
-	 * 
-	 * @param set
-	 *            set spec
-	 * @return set CQL query
-	 */
-	String getSetQuery(final String set);
-
-	/**
-	 * 
-	 * Gets the number of elements in the set with the given set spec for the given metadata prefix.
-	 * 
-	 * @param setSpec
-	 * @param mdPrefix
-	 * @return -1 if the number of elements is unknown, 0 if the set is empty, a positive number otherwise.
-	 */
-	int count(final String setSpec, final String mdPrefix);
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/SetInfo.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/SetInfo.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/SetInfo.java	(nonexistent)
@@ -1,99 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai.utils;
-
-public class SetInfo {
-
-	private String setSpec;
-	private String setName;
-	private String setDescription;
-	private String query;
-	private boolean enabled;
-
-	public SetInfo() {}
-
-	public SetInfo(final String spec, final String name, final String description, final boolean enabled) {
-		this.setSpec = spec;
-		this.setName = name;
-		this.setDescription = description;
-		this.enabled = enabled;
-	}
-
-	/**
-	 * This is a string-only constructor to be used by the SplittedQueryExecutor, which enables us to create instances of this class
-	 * directly from the information in the IS.
-	 * 
-	 * @param spec
-	 * @param name
-	 * @param description
-	 * @param enabled
-	 */
-	public SetInfo(final String spec, final String name, final String description, final String enabled) {
-		this(spec, name, description, Boolean.valueOf(enabled));
-	}
-
-	public SetInfo(final String spec, final String name, final String description, final String query, final boolean enabled) {
-		this(spec, name, description, enabled);
-		this.query = query;
-	}
-
-	/**
-	 * This is a string-only constructor to be used by the SplittedQueryExecutor, which enables us to create instances of this class
-	 * directly from the information in the IS.
-	 * 
-	 * @param spec
-	 * @param name
-	 * @param description
-	 * @param query
-	 * @param enabled
-	 */
-	public SetInfo(final String spec, final String name, final String description, final String query, final String enabled) {
-		this(spec, name, description, enabled);
-		this.query = query;
-	}
-
-	public String getSetSpec() {
-		return setSpec;
-	}
-
-	public void setSetSpec(final String setSpec) {
-		this.setSpec = setSpec;
-	}
-
-	public String getSetDescription() {
-		return setDescription;
-	}
-
-	public void setSetDescription(final String setDescription) {
-		this.setDescription = setDescription;
-	}
-
-	public String getQuery() {
-		return query;
-	}
-
-	public void setQuery(final String query) {
-		this.query = query;
-	}
-
-	public String getSetName() {
-		return setName;
-	}
-
-	public void setSetName(final String setName) {
-		this.setName = setName;
-	}
-
-	public boolean isEnabled() {
-		return enabled;
-	}
-
-	public void setEnabled(final boolean enabled) {
-		this.enabled = enabled;
-	}
-
-	@Override
-	public String toString() {
-		return "SetInfo [setSpec=" + setSpec + ", setName=" + setName + ", setDescription=" + setDescription + ", query=" + query + ", enabled=" + enabled
-				+ "]";
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/MongoSetCollection.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/MongoSetCollection.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/utils/MongoSetCollection.java	(nonexistent)
@@ -1,188 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai.utils;
-
-import java.text.Normalizer;
-import java.util.List;
-
-import org.apache.commons.lang.StringEscapeUtils;
-import org.apache.commons.lang.StringUtils;
-
-import com.google.common.collect.Lists;
-import com.mongodb.BasicDBObject;
-import com.mongodb.BasicDBObjectBuilder;
-import com.mongodb.DB;
-import com.mongodb.DBCollection;
-import com.mongodb.DBCursor;
-import com.mongodb.DBObject;
-
-public class MongoSetCollection implements SetCollection {
-
-	public static String DEFAULT_SET = "OTHER";
-	private DB publisherDB;
-	private String setCollection = "sets";
-	private String setCountCollection = "setsCount";
-
-	public MongoSetCollection(final DB mongoDB) {
-		this.publisherDB = mongoDB;
-	}
-
-	public void ensureIndexes() {
-		this.ensureIndexesOnSets();
-		this.ensureIndexesOnCount();
-	}
-
-	@Override
-	public List<SetInfo> getAllSets() {
-		DBCursor cursor = this.getSetsCollection().find();
-		List<SetInfo> res = Lists.newArrayList();
-		while (cursor.hasNext()) {
-			DBObject obj = cursor.next();
-			res.add(this.getSetFromDBObject(obj));
-		}
-		return res;
-	}
-
-	@Override
-	public boolean containSet(final String set) {
-		DBObject query = new BasicDBObject("spec", set);
-		return this.getSetsCollection().find(query).limit(1).size() != 0;
-	}
-
-	@Override
-	public String getSetQuery(final String set) {
-		DBObject query = new BasicDBObject("spec", set);
-		DBObject returnField = new BasicDBObject("query", 1);
-		DBObject obj = this.getSetsCollection().findOne(query, returnField);
-		return (String) obj.get("query");
-	}
-
-	@Override
-	public int count(final String setSpec, final String mdPrefix) {
-		DBObject query = BasicDBObjectBuilder.start("spec", setSpec).add("mdPrefix", mdPrefix).get();
-		DBObject returnField = new BasicDBObject("count", 1);
-		DBObject obj = this.getSetsCountCollection().findOne(query, returnField);
-		return (Integer) obj.get("count");
-	}
-
-	public void updateCounts(final String setSpec, final String mdPrefix, final int count) {
-		DBObject setCount = BasicDBObjectBuilder.start("spec", setSpec).add("mdPrefix", mdPrefix).get();
-		this.getSetsCountCollection().update(setCount, new BasicDBObject("$set", new BasicDBObject("count", count)), true, false);
-	}
-
-	public void upsertSet(final SetInfo setInfo, final boolean fromConfiguration) {
-		DBObject obj = this.getObjectFromSet(setInfo);
-		obj.put("fromConfiguration", fromConfiguration);
-		this.getSetsCollection().update(new BasicDBObject("spec", setInfo.getSetSpec()), obj, true, false);
-	}
-
-	public String normalizeSetSpec(final String setName) {
-		String s = StringEscapeUtils.unescapeXml(setName);
-		s = Normalizer.normalize(s, Normalizer.Form.NFD);
-		// replace spaces with underscores
-		s = s.replaceAll(" ", "_");
-		// remove tilde, dots... over letters
-		s = s.replaceAll("[\\p{InCombiningDiacriticalMarks}&&[^-_]]", "");
-		// change punctuation into an underscore
-		s = s.replaceAll("[\\p{Punct}&&[^-_]]", "_");
-		// remove all non-word characters
-		s = s.replaceAll("[\\W&&[^-_]]", "");
-		// Avoiding set '___' generated when we have "strange" set names such as those in cyrillic/ukrain
-		// strips _ from the beginning and the end
-		String stripped = StringUtils.strip(s, "_ ");
-		if (StringUtils.isBlank(stripped)) {
-			stripped = DEFAULT_SET;
-		}
-		return stripped;
-	}
-
-	public List<SetInfo> getConfiguredSets() {
-		DBObject query = new BasicDBObject("fromConfiguration", true);
-		return this.findSets(query);
-	}
-
-	public List<SetInfo> getSetsFromData() {
-		DBObject query = new BasicDBObject("fromConfiguration", false);
-		return this.findSets(query);
-	}
-
-	public void dropOAISets() {
-		this.getSetsCountCollection().drop();
-		this.getSetsCollection().drop();
-	}
-
-	public void dropConfigurationSets() {
-		this.getSetsCollection().remove(new BasicDBObject("fromConfiguration", true));
-	}
-
-	protected List<SetInfo> findSets(final DBObject query) {
-		DBCursor cursor = this.getSetsCollection().find(query);
-		List<SetInfo> res = Lists.newArrayList();
-		while (cursor.hasNext()) {
-			DBObject obj = cursor.next();
-			res.add(this.getSetFromDBObject(obj));
-		}
-		return res;
-	}
-
-	private SetInfo getSetFromDBObject(final DBObject obj) {
-		SetInfo setInfo = new SetInfo();
-		setInfo.setEnabled((Boolean) obj.get("enabled"));
-		setInfo.setQuery((String) obj.get("query"));
-		setInfo.setSetDescription((String) obj.get("description"));
-		setInfo.setSetName((String) obj.get("name"));
-		setInfo.setSetSpec((String) obj.get("spec"));
-		return setInfo;
-	}
-
-	private DBObject getObjectFromSet(final SetInfo s) {
-		DBObject obj = BasicDBObjectBuilder.start("spec", s.getSetSpec()).add("name", s.getSetName()).add("description", s.getSetDescription())
-				.add("query", s.getQuery()).add("enabled", s.isEnabled()).get();
-		return obj;
-	}
-
-	private void ensureIndexesOnSets() {
-		this.getSetsCollection().ensureIndex(new BasicDBObject("spec", 1));
-		this.getSetsCollection().ensureIndex(new BasicDBObject("fromConfiguration", 1));
-	}
-
-	private void ensureIndexesOnCount() {
-		DBObject index = BasicDBObjectBuilder.start("spec", 1).add("mdPrefix", 1).get();
-		this.getSetsCountCollection().ensureIndex(index);
-	}
-
-	public DBCollection getSetsCollection() {
-		return this.getCollection(this.setCollection);
-	}
-
-	public DBCollection getSetsCountCollection() {
-		return this.getCollection(this.setCountCollection);
-	}
-
-	private DBCollection getCollection(final String collectionName) {
-		return this.publisherDB.getCollection(collectionName);
-	}
-
-	public DB getPublisherDB() {
-		return publisherDB;
-	}
-
-	public void setPublisherDB(final DB publisherDB) {
-		this.publisherDB = publisherDB;
-	}
-
-	public String getSetCollection() {
-		return setCollection;
-	}
-
-	public void setSetCollection(final String setCollection) {
-		this.setCollection = setCollection;
-	}
-
-	public String getSetCountCollection() {
-		return setCountCollection;
-	}
-
-	public void setSetCountCollection(final String setCountCollection) {
-		this.setCountCollection = setCountCollection;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/OaiFeedMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/OaiFeedMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/oai/OaiFeedMapper.java	(nonexistent)
@@ -1,344 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.oai;
-
-import java.io.IOException;
-import java.net.UnknownHostException;
-import java.text.ParseException;
-import java.util.Collection;
-import java.util.Date;
-import java.util.Map;
-
-import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.io.NullWritable;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Mapper;
-import org.apache.solr.common.util.DateUtil;
-
-import com.google.common.base.Function;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-import com.google.common.collect.Multimap;
-import com.mongodb.BasicDBObject;
-import com.mongodb.DB;
-import com.mongodb.DBCollection;
-import com.mongodb.DBObject;
-import com.mongodb.Mongo;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.hbase.oai.config.OAIConfiguration;
-import eu.dnetlib.data.mapreduce.hbase.oai.config.OAIConfigurationReader;
-import eu.dnetlib.data.mapreduce.hbase.oai.config.OAIConfigurationStringReader;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.MongoSetCollection;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.PublisherField;
-import eu.dnetlib.data.mapreduce.hbase.oai.utils.RecordFieldsExtractor;
-import eu.dnetlib.data.mapreduce.util.OafRowKeyDecoder;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.miscutils.functional.xml.IndentXmlString;
-
-public class OaiFeedMapper extends Mapper<Text, Text, NullWritable, NullWritable> {
-
-	enum RecordStatus {
-		NEW, UPDATED, UNCHANGED;
-	}
-
-	private DBCollection collection;
-	private DBCollection discardedCollection;
-	private OAIConfigurationStringReader oaiConfigurationReader;
-	private OAIConfiguration oaiConfiguration;
-
-	private Date feedDate;
-
-	private MongoSetCollection mongoSetCollection;
-
-	private RecordFieldsExtractor extractor;
-
-	// these are set in the setup
-	private String format;
-	private String interpretation;
-	private String layout;
-	private Map<String, PublisherField> fieldsToIndex = Maps.newHashMap();
-
-	private String duplicateXPath;
-	private boolean skipDuplicates;
-
-	@Override
-	protected void setup(final Context context) throws UnknownHostException {
-
-		String host = context.getConfiguration().get("services.publisher.oai.host");
-		String port = context.getConfiguration().get("services.publisher.oai.port");
-		String db = context.getConfiguration().get("services.publisher.oai.db");
-		String collectionName = context.getConfiguration().get("services.publisher.oai.collection");
-
-		System.out.println("Mongodb client params");
-		System.out.println("host: " + host);
-		System.out.println("port: " + port);
-		System.out.println("db: " + db);
-		System.out.println("collection: " + collectionName);
-
-		String[] formatLayoutInterp = collectionName.split("-");
-		format = formatLayoutInterp[0];
-		layout = formatLayoutInterp[1];
-		interpretation = formatLayoutInterp[2];
-
-		String oaiConfigurationProfile = context.getConfiguration().get("oaiConfiguration");
-		System.out.println("oaiConfiguration:\n" + IndentXmlString.apply(oaiConfigurationProfile));
-		oaiConfigurationReader = new OAIConfigurationStringReader(oaiConfigurationProfile);
-		oaiConfiguration = oaiConfigurationReader.getOaiConfiguration();
-
-		System.out.println("parsed configuration:" + oaiConfiguration.toString());
-
-		Mongo mongo = new Mongo(host, Integer.parseInt(port));
-		DB mongoDB = mongo.getDB(db);
-		collection = mongoDB.getCollection(collectionName);
-		discardedCollection = mongoDB.getCollection("discarded-" + collectionName);
-		mongoSetCollection = new MongoSetCollection(mongoDB);
-
-		duplicateXPath = context.getConfiguration().get("services.publisher.oai.duplicateXPath");
-		skipDuplicates = Boolean.parseBoolean(context.getConfiguration().get("services.publisher.oai.skipDuplicates"));
-
-		Collection<PublisherField> indexFields = oaiConfiguration.getFieldsFor(format, layout, interpretation);
-		extractor = new RecordFieldsExtractor(Lists.newArrayList(indexFields));
-		extractor.setDuplicateXPath(duplicateXPath);
-		extractor.setSkipDuplicates(skipDuplicates);
-
-		for (PublisherField field : indexFields) {
-			fieldsToIndex.put(field.getFieldName(), field);
-		}
-
-		String feedDateString = context.getConfiguration().get(JobParams.OAI_FEED_DATE);
-		try {
-			feedDate = DateUtil.parseDate(feedDateString);
-		} catch (ParseException e) {
-			e.printStackTrace(System.err);
-			throw new RuntimeException(e);
-		}
-
-	}
-
-	@Override
-	protected void map(final Text key, final Text value, final Context context) throws IOException, InterruptedException {
-
-		String recordKey = key.toString();
-		Type entityType = OafRowKeyDecoder.decode(recordKey).getType();
-		switch (entityType) {
-		case person:
-			context.getCounter("oai", "discardedPerson").increment(1);
-			break;
-		default:
-			String recordBody = value.toString();
-			if (StringUtils.isBlank(recordBody)) {
-				discard(context, recordKey, recordBody, "blank body");
-			} else {
-				Multimap<String, String> recordFields = extractor.extractFields(recordBody);
-				String id = "";
-				String oaiID = "";
-				if (checkRecordFields(recordFields, context, recordKey, recordBody)) {
-					id = recordFields.get(OAIConfigurationReader.ID_FIELD).iterator().next();
-					oaiID = getOAIIdentifier(id);
-					handleRecord(getRecordStatus(oaiID, recordBody), context, oaiID, recordBody, recordFields);
-				}
-			}
-		}
-	}
-
-	public boolean checkRecordFields(final Multimap<String, String> recordFields, final Context context, final String recordKey, final String recordBody) {
-		if ((recordFields == null)) {
-			context.getCounter("oai", "invalid").increment(1);
-			return false;
-		}
-		if (recordFields.containsEntry("duplicate", "true")) {
-			if (skipDuplicates) {
-				context.getCounter("oai", "discardedDuplicate").increment(1);
-				return false;
-			} else return true;
-		}
-		if (!recordFields.containsKey(OAIConfigurationReader.ID_FIELD)) {
-			discard(context, recordKey, recordBody, "missing " + OAIConfigurationReader.ID_FIELD);
-			return false;
-		}
-
-		return true;
-
-	}
-
-	private void handleRecord(final RecordStatus status,
-			final Context context,
-			final String oaiID,
-			final String record,
-			final Multimap<String, String> recordProperties) {
-		DBObject obj = null;
-		String counterReason = "";
-		switch (status) {
-		case NEW:
-			obj = this.createBasicObject(oaiID, record, recordProperties, context);
-			obj.put(OAIConfigurationReader.LAST_COLLECTION_DATE_FIELD, feedDate);
-			obj.put(OAIConfigurationReader.DATESTAMP_FIELD, feedDate);
-			obj.put(OAIConfigurationReader.UPDATED_FIELD, false);
-			counterReason = "new record";
-			break;
-		case UPDATED:
-			obj = this.createBasicObject(oaiID, record, recordProperties, context);
-			obj.put(OAIConfigurationReader.LAST_COLLECTION_DATE_FIELD, feedDate);
-			obj.put(OAIConfigurationReader.DATESTAMP_FIELD, feedDate);
-			obj.put(OAIConfigurationReader.UPDATED_FIELD, true);
-			counterReason = "updated record";
-			break;
-		case UNCHANGED:
-			// I just want to set the last collection date, without changing/resetting any of the other dates
-			obj = new BasicDBObject("$set", new BasicDBObject(OAIConfigurationReader.LAST_COLLECTION_DATE_FIELD, feedDate));
-			counterReason = "unchanged record";
-			break;
-		}
-		collection.insert(obj);
-		context.getCounter("oai", counterReason).increment(1);
-		context.getCounter("oai", "total").increment(1);
-	}
-
-	private void discard(final Context context, final String recordKey, final String recordBody, final String reason) {
-		context.getCounter("oai", reason).increment(1);
-		discardedCollection.insert(new BasicDBObject("id", recordKey).append(OAIConfigurationReader.BODY_FIELD, recordBody));
-	}
-
-	private String getOAIIdentifier(final String id) {
-		return oaiConfiguration.getIdScheme() + ":" + oaiConfiguration.getIdNamespace() + ":" + id;
-	}
-
-	private RecordStatus getRecordStatus(final String oaiId, final String recordBody) {
-		if (isNewRecord(oaiId)) return RecordStatus.NEW;
-		if (isChanged(oaiId, recordBody)) return RecordStatus.UPDATED;
-		else return RecordStatus.UNCHANGED;
-	}
-
-	private boolean isNewRecord(final String oaiIdentifier) {
-		// return this.collection.findOne(new BasicDBObject(OAIConfigurationReader.ID_FIELD, oaiIdentifier)) == null;
-		return true;
-	}
-
-	/**
-	 * Tells if record has changed w.r.t. the version we currently have in the store.
-	 * 
-	 * <p>
-	 * FIXME: see MongoPublisherStore and its implementation of the method. For now we just consider that a record has changed and return
-	 * true.
-	 * </p>
-	 * 
-	 * @param oaiID
-	 *            oai identifier of the record
-	 * @param recordBody
-	 *            the record body
-	 * @return true
-	 */
-	private boolean isChanged(final String oaiID, final String recordBody) {
-		return true;
-	}
-
-	protected DBObject createBasicObject(final String oaiID, final String record, final Multimap<String, String> recordProperties, final Context context) {
-		DBObject obj = new BasicDBObject();
-		for (final String key : recordProperties.keySet()) {
-			if (key.equals(OAIConfigurationReader.ID_FIELD)) {
-				obj.put(key, oaiID);
-			} else {
-				Collection<String> values = recordProperties.get(key);
-				if (key.equals(OAIConfigurationReader.SET_FIELD)) {
-
-					Iterable<String> setSpecs = Iterables.transform(values, new Function<String, String>() {
-
-						@Override
-						public String apply(final String s) {
-							return mongoSetCollection.normalizeSetSpec(s);
-						}
-
-					});
-					obj.put(key, setSpecs);
-				} else {
-					PublisherField keyField = fieldsToIndex.get(key);
-					if (keyField == null) {
-						context.getCounter("oai", key + " found for record but not in configuration. Assuming it is repeatable.").increment(1);
-					}
-					// let's check if the key is the name of a repeatable field or not
-					if ((keyField != null) && !keyField.isRepeatable()) {
-						if ((values != null) && !values.isEmpty()) {
-							obj.put(key, values.iterator().next());
-						}
-					} else {
-						obj.put(key, values);
-					}
-				}
-			}
-		}
-		obj.put(OAIConfigurationReader.BODY_FIELD, record);
-		obj.put(OAIConfigurationReader.DELETED_FIELD, false);
-		return obj;
-	}
-
-	@Override
-	protected void cleanup(final Context context) throws IOException, InterruptedException {
-
-		super.cleanup(context);
-	}
-
-	public DBCollection getCollection() {
-		return collection;
-	}
-
-	public void setCollection(final DBCollection collection) {
-		this.collection = collection;
-	}
-
-	public DBCollection getDiscardedCollection() {
-		return discardedCollection;
-	}
-
-	public void setDiscardedCollection(final DBCollection discardedCollection) {
-		this.discardedCollection = discardedCollection;
-	}
-
-	public OAIConfigurationStringReader getOaiConfigurationReader() {
-		return oaiConfigurationReader;
-	}
-
-	public void setOaiConfigurationReader(final OAIConfigurationStringReader oaiConfigurationReader) {
-		this.oaiConfigurationReader = oaiConfigurationReader;
-	}
-
-	public OAIConfiguration getOaiConfiguration() {
-		return oaiConfiguration;
-	}
-
-	public void setOaiConfiguration(final OAIConfiguration oaiConfiguration) {
-		this.oaiConfiguration = oaiConfiguration;
-	}
-
-	public Date getFeedDate() {
-		return feedDate;
-	}
-
-	public void setFeedDate(final Date feedDate) {
-		this.feedDate = feedDate;
-	}
-
-	public MongoSetCollection getMongoSetCollection() {
-		return mongoSetCollection;
-	}
-
-	public void setMongoSetCollection(final MongoSetCollection mongoSetCollection) {
-		this.mongoSetCollection = mongoSetCollection;
-	}
-
-	public String getDuplicateXPath() {
-		return duplicateXPath;
-	}
-
-	public void setDuplicateXPath(final String duplicateXPath) {
-		this.duplicateXPath = duplicateXPath;
-	}
-
-	public boolean isSkipDuplicates() {
-		return skipDuplicates;
-	}
-
-	public void setSkipDuplicates(final boolean skipDuplicates) {
-		this.skipDuplicates = skipDuplicates;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataexport/HBaseCopyTableJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataexport/HBaseCopyTableJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataexport/HBaseCopyTableJob.java	(nonexistent)
@@ -1,35 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dataexport;
-
-import java.util.Properties;
-
-import org.apache.hadoop.hbase.mapreduce.CopyTable;
-import org.apache.hadoop.mapreduce.Job;
-
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-
-import eu.dnetlib.data.mapreduce.hbase.AbstractHBaseMapReduceJob;
-
-/**
- * HBaseCopyTableJob is a simple wrapper over org.apache.hadoop.hbase.mapreduce.CopyTable that makes it invokable from a
- * D-Net workflow, passing the required parameters.
- * 
- * Copies the given table to the remote hbase instance.
- * 
- * @author claudio
- * 
- */
-public class HBaseCopyTableJob extends AbstractHBaseMapReduceJob {
-
-	@Override
-	protected Job setJobDetails(Job job, Properties p) throws Exception {
-
-		String remoteCluster = "--peer.adr=" + p.getProperty("peername");
-		String tablename = p.getProperty("tablename");
-
-		String[] args = Iterables.toArray(Lists.newArrayList(remoteCluster, tablename), String.class);
-
-		return CopyTable.createSubmittableJob(job.getConfiguration(), args);
-	}
-
-}

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataexport/HBaseCopyTableJob.java
___________________________________________________________________
Deleted: svn:mime-type
## -1 +0,0 ##
-text/plain
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/HBaseTableUtils.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/HBaseTableUtils.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/HBaseTableUtils.java	(nonexistent)
@@ -1,96 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase;
-
-import java.math.BigInteger;
-import java.util.List;
-import java.util.Set;
-
-import com.google.common.base.Function;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Sets;
-
-import eu.dnetlib.data.proto.RelTypeProtos.RelType;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-
-/**
- * Common static utility methods to manage the hbase tables
- * 
- * @author claudio
- * 
- */
-public class HBaseTableUtils {
-
-	public final static String lowerKey = "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000";
-	public final static String upperKey = "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz";
-
-	public enum VolatileColumnFamily {
-		dedup, dedupPerson; // instance is here to remove the old protos
-
-		public static boolean isVolatile(String columnName) {
-			try {
-				return VolatileColumnFamily.valueOf(columnName) != null;
-			} catch (Throwable e) {
-				return false;
-			}
-		}
-	}
-
-	private static Function<Type, String> typeName = new Function<Type, String>() {
-
-		@Override
-		public String apply(Type type) {
-			return type.toString();
-		}
-	};
-
-	private static Function<RelType, String> relTypeName = new Function<RelType, String>() {
-
-		@Override
-		public String apply(RelType type) {
-			return type.toString();
-		}
-	};
-
-	private static Function<VolatileColumnFamily, String> extraColumnFamilyName = new Function<VolatileColumnFamily, String>() {
-
-		@Override
-		public String apply(VolatileColumnFamily type) {
-			return type.toString();
-		}
-	};
-
-	public static Set<String> listColumns() {
-		return Sets.newHashSet(Iterables.concat(Iterables.transform(Lists.newArrayList(Type.values()), typeName),
-				Iterables.transform(Lists.newArrayList(RelType.values()), relTypeName),
-				Iterables.transform(Lists.newArrayList(VolatileColumnFamily.values()), extraColumnFamilyName)));
-	}
-
-	public static byte[][] getHexSplits(List<String> splitKeys) {
-		byte[][] splits = new byte[splitKeys.size() - 1][];
-		for (int i = 0; i < splitKeys.size() - 1; i++) {
-			byte[] b = splitKeys.get(i).getBytes();
-			splits[i] = b;
-		}
-		return splits;
-	}
-
-	public static byte[][] getHexSplits(int numRegions) {
-		return getHexSplits(lowerKey, upperKey, numRegions);
-	}
-
-	public static byte[][] getHexSplits(String startKey, String endKey, int numRegions) {
-		byte[][] splits = new byte[numRegions - 1][];
-		BigInteger lowestKey = new BigInteger(startKey, 16);
-		BigInteger highestKey = new BigInteger(endKey, 16);
-		BigInteger range = highestKey.subtract(lowestKey);
-		BigInteger regionIncrement = range.divide(BigInteger.valueOf(numRegions));
-		lowestKey = lowestKey.add(regionIncrement);
-		for (int i = 0; i < numRegions - 1; i++) {
-			BigInteger key = lowestKey.add(regionIncrement.multiply(BigInteger.valueOf(i)));
-			byte[] b = String.format("%016x", key).getBytes();
-			splits[i] = b;
-		}
-		return splits;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/stats/StatsFeedMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/stats/StatsFeedMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/stats/StatsFeedMapper.java	(nonexistent)
@@ -1,81 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.stats;
-
-import java.io.IOException;
-import java.util.List;
-
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Mapper;
-
-import com.google.common.collect.Lists;
-
-import eu.dnetlib.miscutils.datetime.HumanTime;
-
-public class StatsFeedMapper extends Mapper<Text, Text, Text, Text> {
-
-	private List<String> buffer;
-
-	private int bufferFlushThreshold = 1000;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		buffer = Lists.newArrayList();
-	}
-
-	@Override
-	protected void map(final Text key, final Text value, final Context context) throws IOException, InterruptedException {
-
-		final String copyCmd = buildCopyCommand(value);
-		try {
-
-			if ((copyCmd != null) && !copyCmd.isEmpty()) {
-
-				buffer.add(copyCmd);
-				if (buffer.size() >= bufferFlushThreshold) {
-					doAdd(buffer, context);
-				}
-			} else {
-				context.getCounter("stats", "skipped records").increment(1);
-			}
-		} catch (Throwable e) {
-			context.getCounter("stats", e.getClass().toString()).increment(1);
-			context.write(key, printRottenRecord(context.getTaskAttemptID().toString(), value, copyCmd));
-			e.printStackTrace(System.err);
-		}
-	}
-
-	private String buildCopyCommand(final Text value) {
-
-		return null;
-	}
-
-	private void doAdd(final List<String> buffer, final Context context) {
-
-		long start = System.currentTimeMillis();
-
-		// flush the buffer
-		// UpdateResponse rsp = serverPool.addAll(buffer);
-		long stop = System.currentTimeMillis() - start;
-		System.out.println("feed time for " + buffer.size() + " records : " + HumanTime.exactly(stop) + "\n");
-
-		buffer.clear();
-	}
-
-	@Override
-	protected void cleanup(final Context context) throws IOException, InterruptedException {
-		super.cleanup(context);
-
-		if (!buffer.isEmpty()) {
-			doAdd(buffer, context);
-		}
-	}
-
-	private Text printRottenRecord(final String taskid, final Text value, final String copyCmq) {
-		return new Text("\n**********************************\n" + "task: " + taskid + "\n" + check("original", value.toString() + check("copy cmd", copyCmq)));
-	}
-
-	private String check(final String label, final Object value) {
-		if ((value != null) && !value.toString().isEmpty()) { return "\n " + label + ":\n" + value + "\n"; }
-		return "\n";
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/stats/StatsFeedJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/stats/StatsFeedJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/stats/StatsFeedJob.java	(nonexistent)
@@ -1,42 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.stats;
-
-import java.io.IOException;
-import java.util.Properties;
-
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
-import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
-import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.hbase.AbstractHBaseMapReduceJob;
-
-public class StatsFeedJob extends AbstractHBaseMapReduceJob {
-
-	@Override
-	public Job setJobDetails(final Job job, final Properties p) {
-
-		job.setInputFormatClass(SequenceFileInputFormat.class);
-		try {
-			FileInputFormat.setInputPaths(job, p.getProperty(JobParams.MAPRED_INPUT_DIR));
-			FileOutputFormat.setOutputPath(job, new Path(p.getProperty(JobParams.STATS_ROTTEN_FILE)));
-		} catch (IOException e) {
-			throw new RuntimeException(e);
-		}
-		job.setMapperClass(StatsFeedMapper.class);
-		job.setMapOutputKeyClass(Text.class);
-		job.setMapOutputValueClass(Text.class);
-
-		job.setNumReduceTasks(0);
-
-		job.getConfiguration().setBoolean("mapred.map.tasks.speculative.execution", false);
-		job.getConfiguration().setBoolean("mapreduce.map.speculative", false);
-
-		job.getConfiguration().setBoolean("mapred.compress.map.output", true);
-
-		return job;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/DeleteActionsJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/DeleteActionsJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/DeleteActionsJob.java	(nonexistent)
@@ -1,30 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.actions;
-
-import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Scan;
-import org.apache.hadoop.hbase.filter.FilterList;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-
-public class DeleteActionsJob extends AbstractActionsJob {
-
-	@Override
-	protected Class<? extends TableMapper<ImmutableBytesWritable, ?>> getMapperClass() {
-		return DeleteActionsMapper.class;
-	}
-
-	@Override
-	protected Class<?> getMapOutputValueClass() {
-		return Delete.class;
-	}
-
-	@Override
-	protected void addSpecificFilters(FilterList filters) {
-		// NOT necessary
-	}
-
-	@Override
-	protected void addSpecificScanClauses(Scan scan) {
-		// NOT necessary
-	}
-}
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/CommitActionsMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/CommitActionsMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/CommitActionsMapper.java	(nonexistent)
@@ -1,46 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.actions;
-
-import java.io.IOException;
-
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-
-public class CommitActionsMapper extends TableMapper<ImmutableBytesWritable, Put> {
-
-	private String set = null;
-
-	@Override
-	protected void setup(Context context) throws IOException, InterruptedException {
-		set = context.getConfiguration().get("set");
-		if (set != null) {
-			System.out.println("Committing set: " + set);
-		} else {
-			System.out.println("Committing ALL sets");
-		}
-	}
-
-	@Override
-	protected void map(ImmutableBytesWritable key, Result value, Context context) throws IOException, InterruptedException {
-
-		if (set == null || value.getColumnLatest(Bytes.toBytes("set"), Bytes.toBytes(set)) != null) {
-			System.out.println("Processing action " + Bytes.toString(key.copyBytes()));
-
-			byte[] cf = Bytes.toBytes("target");
-
-			byte[] tkey = value.getValue(cf, Bytes.toBytes("rowKey"));
-			byte[] tcf = value.getValue(cf, Bytes.toBytes("columnFamily"));
-			byte[] tc = value.getValue(cf, Bytes.toBytes("column"));
-			byte[] tv = value.getValue(cf, Bytes.toBytes("content"));
-
-			Put put = new Put(tkey);
-			put.add(tcf, tc, tv);
-
-			context.getCounter("Actions", Bytes.toString(tcf)).increment(1);
-
-			context.write(new ImmutableBytesWritable(tkey), put);
-		}
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/AbstractActionsJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/AbstractActionsJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/AbstractActionsJob.java	(nonexistent)
@@ -1,75 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.actions;
-
-import java.io.IOException;
-import java.util.Properties;
-
-import org.apache.hadoop.hbase.client.Scan;
-import org.apache.hadoop.hbase.filter.FilterList;
-import org.apache.hadoop.hbase.filter.FilterList.Operator;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.mapreduce.TableOutputFormat;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Job;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.hbase.AbstractHBaseMapReduceJob;
-
-abstract public class AbstractActionsJob extends AbstractHBaseMapReduceJob {
-
-	@Override
-	protected Job setJobDetails(Job job, Properties p) throws Exception {
-		initMapper(job, getScan(p), p.getProperty(JobParams.HBASE_SOURCE_TABLE));
-
-		job.setOutputFormatClass(TableOutputFormat.class);
-		job.setMapOutputKeyClass(Text.class);
-		job.setMapOutputValueClass(getMapOutputValueClass());
-		job.setNumReduceTasks(0);
-
-		return job;
-	}
-
-	private void initMapper(final Job job, final Scan scan, final String sourceTable) {
-		try {
-			TableMapReduceUtil.initTableMapperJob(sourceTable, scan, getMapperClass(), Text.class, ImmutableBytesWritable.class, job);
-		} catch (IOException e) {
-			throw new RuntimeException(e);
-		}
-	}
-
-	abstract protected Class<? extends TableMapper<ImmutableBytesWritable, ?>> getMapperClass();
-
-	abstract protected Class<?> getMapOutputValueClass();
-
-	abstract protected void addSpecificFilters(FilterList filters);
-
-	abstract protected void addSpecificScanClauses(Scan scan);
-
-	private Scan getScan(Properties p) {
-
-		Scan scan = new Scan();
-		scan.setCaching(500);
-		scan.setCacheBlocks(false);
-
-		FilterList filters = new FilterList(Operator.MUST_PASS_ALL);
-
-		//if (p.containsKey("set")) {
-		//filters.addFilter(new FamilyFilter(CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes("set"))));
-		//filters.addFilter(new QualifierFilter(CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes(p.getProperty("set")))));
-		//scan.addFamily(Bytes.toBytes("set"));
-		//			byte[] qualifier = Bytes.toBytes(p.getProperty("set"));
-		//			SingleColumnValueFilter filter = new SingleColumnValueFilter(Bytes.toBytes("set"), qualifier, CompareOp.EQUAL, qualifier);
-		//			filter.setFilterIfMissing(true);
-		//			filters.addFilter(filter);
-		//scan.addColumn(Bytes.toBytes("set"), Bytes.toBytes(p.getProperty("set")));
-		//}
-
-		addSpecificFilters(filters);
-		addSpecificScanClauses(scan);
-
-		scan.setFilter(filters);
-		return scan;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/CommitActionsJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/CommitActionsJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/CommitActionsJob.java	(nonexistent)
@@ -1,32 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.actions;
-
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.client.Scan;
-import org.apache.hadoop.hbase.filter.FilterList;
-import org.apache.hadoop.hbase.filter.PrefixFilter;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-
-public class CommitActionsJob extends AbstractActionsJob {
-
-	@Override
-	protected Class<? extends TableMapper<ImmutableBytesWritable, ?>> getMapperClass() {
-		return CommitActionsMapper.class;
-	}
-
-	@Override
-	protected Class<?> getMapOutputValueClass() {
-		return Put.class;
-	}
-
-	@Override
-	protected void addSpecificFilters(FilterList filters) {
-		filters.addFilter(new PrefixFilter(Bytes.toBytes("aac|")));
-	}
-
-	@Override
-	protected void addSpecificScanClauses(Scan scan) {
-		//scan.addFamily(Bytes.toBytes("target"));
-	}
-}
\ No newline at end of file
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/DeleteActionsMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/DeleteActionsMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions/DeleteActionsMapper.java	(nonexistent)
@@ -1,36 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.actions;
-
-import java.io.IOException;
-
-import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-
-public class DeleteActionsMapper extends TableMapper<ImmutableBytesWritable, Delete> {
-
-	private String set = null;
-
-	@Override
-	protected void setup(Context context) throws IOException, InterruptedException {
-		set = context.getConfiguration().get("set");
-		if (set != null) {
-			System.out.println("Deleting set: " + set);
-		} else {
-			System.out.println("Deleting ALL sets");
-		}
-	}
-
-	@Override
-	protected void map(ImmutableBytesWritable key, Result value, Context context) throws IOException, InterruptedException {
-		if (set == null || value.getColumnLatest(Bytes.toBytes("set"), Bytes.toBytes(set)) != null) {
-
-			final byte[] rowKey = key.copyBytes();
-			System.out.println("Deleting action " + Bytes.toString(rowKey));
-
-			context.getCounter("Actions", "N. Deletes").increment(1);
-			context.write(key, new Delete(rowKey));
-		}
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/FindPersonCoauthorsReducer.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/FindPersonCoauthorsReducer.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/FindPersonCoauthorsReducer.java	(nonexistent)
@@ -1,5 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-public class FindPersonCoauthorsReducer {
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupGrouperMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupGrouperMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupGrouperMapper.java	(nonexistent)
@@ -1,64 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.util.Arrays;
-import java.util.List;
-
-import org.apache.hadoop.hbase.KeyValue;
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.util.DedupUtils;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.pace.config.DedupConfig;
-
-public class DedupGrouperMapper extends TableMapper<Text, Put> {
-
-	private static final boolean WRITE_TO_WAL = false;
-
-	public static final String COUNTER_GROUP = "dedup.grouper";
-
-	public static final String COUNTER_NAME = "written.rels";
-
-	private Text rowKey;
-
-	private DedupConfig dedupConf;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		rowKey = new Text();
-
-		dedupConf = DedupConfig.load(context.getConfiguration().get(JobParams.DEDUP_CONF));
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable keyIn, final Result value, final Context context) throws IOException, InterruptedException {
-
-		final List<KeyValue> kvList = value.list();
-		// System.out.println("Grouper mapping " + kvList.size() + " rels for key: " + new String(keyIn.copyBytes()));
-
-		for (final KeyValue n : kvList) {
-			for (final KeyValue j : kvList) {
-
-				final byte[] nq = n.getQualifier();
-				final byte[] jq = j.getQualifier();
-
-				if (!Arrays.equals(nq, jq)) {
-
-					final Put put = new Put(nq).add(DedupUtils.getSimilarityCFBytes(Type.valueOf(dedupConf.getWf().getEntityType())), jq, Bytes.toBytes(""));
-					put.setWriteToWAL(WRITE_TO_WAL);
-					rowKey.set(nq);
-					context.write(rowKey, put);
-
-					context.getCounter(COUNTER_GROUP, COUNTER_NAME).increment(1);
-				}
-			}
-		}
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupReducer.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupReducer.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupReducer.java	(nonexistent)
@@ -1,207 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.util.HashSet;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.PriorityQueue;
-import java.util.Queue;
-import java.util.Set;
-
-import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableReducer;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-
-import com.google.common.collect.Lists;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.util.DedupUtils;
-import eu.dnetlib.data.proto.RelTypeProtos.SubRelType;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.pace.clustering.NGramUtils;
-import eu.dnetlib.pace.config.DedupConfig;
-import eu.dnetlib.pace.distance.PaceDocumentDistance;
-import eu.dnetlib.pace.model.FieldList;
-import eu.dnetlib.pace.model.MapDocument;
-import eu.dnetlib.pace.model.MapDocumentComparator;
-import eu.dnetlib.pace.model.MapDocumentSerializer;
-
-public class DedupReducer extends TableReducer<Text, ImmutableBytesWritable, ImmutableBytesWritable> {
-
-	private static final boolean WRITE_TO_WAL = false;
-
-	private DedupConfig dedupConf;
-
-	private ImmutableBytesWritable ibw;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-
-		dedupConf = DedupConfig.load(context.getConfiguration().get(JobParams.DEDUP_CONF));
-		ibw = new ImmutableBytesWritable();
-
-		System.out.println("dedup reduce phase \npace conf: " + dedupConf.toString());
-	}
-
-	@Override
-	protected void reduce(final Text key, final Iterable<ImmutableBytesWritable> values, final Context context) throws IOException, InterruptedException {
-		System.out.println("\nReducing key: '" + key + "'");
-
-		final Queue<MapDocument> q = prepare(context, key, values);
-		switch (Type.valueOf(dedupConf.getWf().getEntityType())) {
-		case person:
-			process(q, context);
-			break;
-		case result:
-			process(simplifyQueue(q, key.toString(), context), context);
-			break;
-		case organization:
-			process(q, context);
-			break;
-		default:
-			throw new IllegalArgumentException("dedup not implemented for type: " + dedupConf.getWf().getEntityType());
-		}
-	}
-
-	private Queue<MapDocument> prepare(final Context context, final Text key, final Iterable<ImmutableBytesWritable> values) {
-		final Queue<MapDocument> queue = new PriorityQueue<MapDocument>(100, new MapDocumentComparator(dedupConf.getWf().getOrderField()));
-
-		final Set<String> seen = new HashSet<String>();
-
-		for (final ImmutableBytesWritable i : values) {
-			final MapDocument doc = MapDocumentSerializer.decode(i.copyBytes());
-			final String id = doc.getIdentifier();
-
-			if (!seen.contains(id)) {
-				seen.add(id);
-				queue.add(doc);
-			}
-
-			if (queue.size() > dedupConf.getWf().getQueueMaxSize()) {
-				// context.getCounter("ngram size > " + LIMIT, "'" + key.toString() + "', --> " + context.getTaskAttemptID()).increment(1);
-				context.getCounter("ngram size > " + dedupConf.getWf().getQueueMaxSize(), "N").increment(1);
-				System.out.println("breaking out after limit (" + dedupConf.getWf().getQueueMaxSize() + ") for ngram '" + key + "'");
-				break;
-			}
-		}
-
-		return queue;
-	}
-
-	private Queue<MapDocument> simplifyQueue(final Queue<MapDocument> queue, final String ngram, final Context context) {
-		final Queue<MapDocument> q = new LinkedList<MapDocument>();
-
-		String fieldRef = "";
-		final List<MapDocument> tempResults = Lists.newArrayList();
-
-		while (!queue.isEmpty()) {
-			final MapDocument result = queue.remove();
-
-			if (!result.values(dedupConf.getWf().getOrderField()).isEmpty()) {
-				final String field = NGramUtils.cleanupForOrdering(result.values(dedupConf.getWf().getOrderField()).stringValue());
-				if (field.equals(fieldRef)) {
-					tempResults.add(result);
-				} else {
-					populateSimplifiedQueue(q, tempResults, context, fieldRef, ngram);
-					tempResults.clear();
-					tempResults.add(result);
-					fieldRef = field;
-				}
-			} else {
-				context.getCounter(dedupConf.getWf().getEntityType(), "missing " + dedupConf.getWf().getOrderField()).increment(1);
-			}
-		}
-		populateSimplifiedQueue(q, tempResults, context, fieldRef, ngram);
-
-		return q;
-	}
-
-	private void populateSimplifiedQueue(final Queue<MapDocument> q,
-			final List<MapDocument> tempResults,
-			final Context context,
-			final String fieldRef,
-			final String ngram) {
-		if (tempResults.size() < dedupConf.getWf().getGroupMaxSize()) {
-			q.addAll(tempResults);
-		} else {
-			context.getCounter(dedupConf.getWf().getEntityType(),
-					"Skipped records for count(" + dedupConf.getWf().getOrderField() + ") >= " + dedupConf.getWf().getGroupMaxSize())
-					.increment(tempResults.size());
-			System.out.println("Skipped field: " + fieldRef + " - size: " + tempResults.size() + " - ngram: " + ngram);
-		}
-	}
-
-	private void process(final Queue<MapDocument> queue, final Context context) throws IOException, InterruptedException {
-
-		final PaceDocumentDistance algo = new PaceDocumentDistance();
-
-		while (!queue.isEmpty()) {
-
-			final MapDocument pivot = queue.remove();
-			final String idPivot = pivot.getIdentifier();
-
-			final FieldList fieldsPivot = pivot.values(dedupConf.getWf().getOrderField());
-			final String fieldPivot = (fieldsPivot == null) || fieldsPivot.isEmpty() ? null : fieldsPivot.stringValue();
-
-			if (fieldPivot != null) {
-				// System.out.println(idPivot + " --> " + fieldPivot);
-
-				int i = 0;
-				for (final MapDocument curr : queue) {
-					final String idCurr = curr.getIdentifier();
-
-					if (mustSkip(idCurr)) {
-						context.getCounter(dedupConf.getWf().getEntityType(), "skip list").increment(1);
-						break;
-					}
-
-					if (i > dedupConf.getWf().getSlidingWindowSize()) {
-						break;
-					}
-
-					final FieldList fieldsCurr = curr.values(dedupConf.getWf().getOrderField());
-					final String fieldCurr = (fieldsCurr == null) || fieldsCurr.isEmpty() ? null : fieldsCurr.stringValue();
-
-					if (!idCurr.equals(idPivot) && (fieldCurr != null)) {
-
-						final double d = algo.between(pivot, curr, dedupConf);
-
-						if (d >= dedupConf.getWf().getThreshold()) {
-							writeSimilarity(context, idPivot, idCurr);
-							context.getCounter(dedupConf.getWf().getEntityType(), SubRelType.dedupSimilarity.toString() + " (x2)").increment(1);
-						} else {
-							context.getCounter(dedupConf.getWf().getEntityType(), "d < " + dedupConf.getWf().getThreshold()).increment(1);
-						}
-						i++;
-					}
-				}
-			}
-		}
-	}
-
-	private boolean mustSkip(final String idPivot) {
-		return dedupConf.getWf().getSkipList().contains(getNsPrefix(idPivot));
-	}
-
-	private String getNsPrefix(final String id) {
-		return StringUtils.substringBetween(id, "|", "::");
-	}
-
-	private void writeSimilarity(final Context context, final String idPivot, final String id) throws IOException, InterruptedException {
-		final byte[] rowKey = Bytes.toBytes(idPivot);
-		final byte[] target = Bytes.toBytes(id);
-
-		emitRel(context, rowKey, target);
-		emitRel(context, target, rowKey);
-	}
-
-	private void emitRel(final Context context, final byte[] from, final byte[] to) throws IOException, InterruptedException {
-		final Put put = new Put(from).add(DedupUtils.getSimilarityCFBytes(Type.valueOf(dedupConf.getWf().getEntityType())), to, Bytes.toBytes(""));
-		put.setWriteToWAL(WRITE_TO_WAL);
-		ibw.set(from);
-		context.write(ibw, put);
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupRootsToCsvMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupRootsToCsvMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupRootsToCsvMapper.java	(nonexistent)
@@ -1,72 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.nio.charset.Charset;
-import java.util.Map;
-
-import org.apache.commons.collections.MapUtils;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-
-import com.google.common.collect.Iterables;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.util.DedupUtils;
-import eu.dnetlib.data.mapreduce.util.OafDecoder;
-import eu.dnetlib.pace.config.DedupConfig;
-
-public class DedupRootsToCsvMapper extends TableMapper<ImmutableBytesWritable, ImmutableBytesWritable> {
-
-	/**
-	 * logger.
-	 */
-	private static final Log log = LogFactory.getLog(DedupRootsToCsvMapper.class); // NOPMD by marko on 11/24/08 5:02 PM
-
-	private DedupConfig dedupConf;
-
-	private ImmutableBytesWritable key;
-
-	private ImmutableBytesWritable value;
-
-	@Override
-	protected void setup(final Context context) {
-		dedupConf = DedupConfig.load(context.getConfiguration().get(JobParams.DEDUP_CONF));
-		System.out.println("dedup buildRoots mapper\nwf conf: " + dedupConf.toString());
-		key = new ImmutableBytesWritable();
-		value = new ImmutableBytesWritable();
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable rowkey, final Result result, final Context context) throws IOException, InterruptedException {
-
-		if (DedupUtils.isRoot(rowkey)) {
-			context.getCounter(dedupConf.getWf().getEntityType(), "root row skipped").increment(1);
-			return;
-		}
-
-		final Map<byte[], byte[]> entityCf = result.getFamilyMap(Bytes.toBytes(dedupConf.getWf().getEntityType()));
-		if (MapUtils.isEmpty(entityCf) && (entityCf.get(DedupUtils.BODY_B) == null)) {
-			context.getCounter(dedupConf.getWf().getEntityType(), "missing body").increment(1);
-			return;
-		}
-
-		final Map<byte[], byte[]> mergedIn = result.getFamilyMap(DedupUtils.getDedupCF_mergedInBytes(dedupConf.getWf().getEntityType()));
-		if (MapUtils.isEmpty(mergedIn)) {
-			context.getCounter(dedupConf.getWf().getEntityType(), "missing mergedIn relationship").increment(1);
-			return;
-		}
-		final String rootId = new String(Iterables.getOnlyElement(mergedIn.keySet()), Charset.forName("UTF-8"));
-		final byte[] body = entityCf.get(DedupUtils.BODY_B);
-
-		key.set(Bytes.toBytes(rootId));
-		value.set(Bytes.toBytes(Iterables.getOnlyElement(OafDecoder.decode(body).getEntity().getOriginalIdList())));
-		context.write(key, value);
-		context.getCounter(dedupConf.getWf().getEntityType(), "root entity").increment(1);
-
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupBuildRootsMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupBuildRootsMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupBuildRootsMapper.java	(nonexistent)
@@ -1,141 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.util.Map;
-import java.util.Set;
-
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-
-import com.google.common.collect.Iterables;
-import com.google.protobuf.InvalidProtocolBufferException;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.util.DedupUtils;
-import eu.dnetlib.data.mapreduce.util.OafDecoder;
-import eu.dnetlib.data.mapreduce.util.UpdateMerger;
-import eu.dnetlib.data.proto.KindProtos.Kind;
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.proto.OafProtos.OafEntity;
-import eu.dnetlib.data.proto.RelTypeProtos.RelType;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.data.transform.OafUtils;
-import eu.dnetlib.pace.config.DedupConfig;
-
-public class DedupBuildRootsMapper extends TableMapper<Text, ImmutableBytesWritable> {
-
-	private DedupConfig dedupConf;
-
-	private ImmutableBytesWritable ibw;
-
-	private Set<String> entityNames;
-
-	@Override
-	protected void setup(final Context context) {
-		dedupConf = DedupConfig.load(context.getConfiguration().get(JobParams.DEDUP_CONF));
-		System.out.println("dedup buildRoots mapper\nwf conf: " + dedupConf.toString());
-
-		entityNames = OafUtils.entities();
-		ibw = new ImmutableBytesWritable();
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable rowkey, final Result value, final Context context) throws IOException, InterruptedException {
-		// System.out.println("Find root mapping: " + new String(rowkey.copyBytes()));
-
-		if (!DedupUtils.isRoot(rowkey)) {
-
-			// TODO: remove this hack - here because we don't want to dedup datasets
-			if (checkDataset(value)) return;
-
-			final Map<byte[], byte[]> dedupRels = value.getFamilyMap(DedupUtils.getDedupCF_mergedInBytes(Type.valueOf(dedupConf.getWf().getEntityType())));
-
-			if ((dedupRels != null) && !dedupRels.isEmpty()) {
-
-				final Text rootId = findRoot(dedupRels);
-				// byte[] rootIdBytes = rootId.copyBytes();
-				// byte[] rowkeyBytes = rowkey.copyBytes();
-
-				context.getCounter(dedupConf.getWf().getEntityType(), "merged").increment(1);
-				for (final String family : dedupConf.getWf().getRootBuilder()) {
-
-					// if (checkHack(rowkeyBytes, rootIdBytes, family)) {
-					// context.getCounter("hack", "personResult skipped").increment(1);
-					// } else {
-
-					final Map<byte[], byte[]> map = value.getFamilyMap(Bytes.toBytes(family));
-					if ((map != null) && !map.isEmpty()) {
-
-						// if we're dealing with the entity CF
-						if (entityNames.contains(family)) {
-							final Oaf body = UpdateMerger.mergeBodyUpdates(context, map);
-
-							emit(context, rootId, body.toByteArray());
-						} else {
-							for (final byte[] o : map.values()) {
-
-								if (!isRelMarkedDeleted(context, o)) {
-									emit(context, rootId, o);
-								} else {
-									context.getCounter(family, "rel marked deleted").increment(1);
-								}
-							}
-						}
-					} // else {
-					// System.err.println("empty family: " + family + "\nkey: " + sKey);
-					// context.getCounter("DedupBuildRootsMapper", "empty family '" + family + "'").increment(1);
-					// }
-					// }
-				}
-			} else {
-				context.getCounter(dedupConf.getWf().getEntityType(), "not in duplicate group").increment(1);
-			}
-		} else {
-			context.getCounter(dedupConf.getWf().getEntityType(), "root").increment(1);
-		}
-	}
-
-	private boolean checkDataset(final Result value) {
-		final Map<byte[], byte[]> bodyMap = value.getFamilyMap(dedupConf.getWf().getEntityType().getBytes());
-
-		if ((bodyMap == null) || bodyMap.isEmpty()) return true;
-
-		final byte[] bodyB = bodyMap.get(DedupUtils.BODY_B);
-
-		if (bodyB == null) return true;
-
-		final OafEntity entity = OafDecoder.decode(bodyB).getEntity();
-
-		if (entity.getType().equals(Type.result) && entity.getResult().getMetadata().getResulttype().getClassid().equals("dataset")) return true;
-
-		return false;
-	}
-
-	private void emit(final Context context, final Text rootId, final byte[] value) throws IOException, InterruptedException {
-		ibw.set(value);
-		context.write(rootId, ibw);
-	}
-
-	private boolean checkHack(final byte[] rowkey, final byte[] rootId, final String family) {
-		return dedupConf.getWf().getEntityType().equals(Type.result.toString()) && 	// we're deduplicating the results
-				family.equals(RelType.personResult.toString()) && 			// we're dealing with a personResult relation
-				!rootId.equals(DedupUtils.newIdBytes(new String(rowkey), dedupConf.getWf().getDedupRun()));
-	}
-
-	private boolean isRelMarkedDeleted(final Context context, final byte[] o) {
-		try {
-			final Oaf oaf = Oaf.parseFrom(o);
-			return oaf.getKind().equals(Kind.relation) && oaf.getDataInfo().getDeletedbyinference();
-		} catch (final InvalidProtocolBufferException e) {
-			context.getCounter("error", e.getClass().getName()).increment(1);
-			return true;
-		}
-	}
-
-	private Text findRoot(final Map<byte[], byte[]> dedupRels) {
-		return new Text(Iterables.getOnlyElement(dedupRels.keySet()));
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/FindDedupCandidatePersonsMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/FindDedupCandidatePersonsMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/FindDedupCandidatePersonsMapper.java	(nonexistent)
@@ -1,60 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.util.Set;
-
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-
-import com.google.common.base.Function;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Sets;
-
-import eu.dnetlib.data.mapreduce.util.DedupUtils;
-import eu.dnetlib.data.mapreduce.util.OafDecoder;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.pace.model.PersonComparatorUtils;
-
-public class FindDedupCandidatePersonsMapper extends TableMapper<Text, Text> {
-
-	private static final byte[] PERSON_CF = Type.person.toString().getBytes();
-	private static final byte[] PERSONRESULT_CF = "personResult".getBytes();
-
-	@Override
-	protected void setup(final Context context) {
-
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable rowkey, final Result row, final Context context) throws IOException, InterruptedException {
-		String id = Bytes.toString(rowkey.get());
-		String fullname = extractFullname(row);
-		Set<String> resultIds = extractResultIds(row);
-
-		if (fullname != null) {
-			Text text = (new DedupPersonBean(id, fullname, resultIds)).toText();
-			for (String k : PersonComparatorUtils.getNgramsForPerson(fullname)) {
-				context.write(new Text(k), text);
-			}
-		}
-	}
-
-	private Set<String> extractResultIds(final Result row) {
-		return Sets.newHashSet(Iterables.transform(row.getFamilyMap(PERSONRESULT_CF).keySet(), new Function<byte[], String>() {
-
-			@Override
-			public String apply(final byte[] b) {
-				return Bytes.toString(b);
-			}
-		}));
-	}
-
-	private String extractFullname(final Result row) {
-		byte[] body = row.getValue(PERSON_CF, DedupUtils.BODY_B);
-		if (body == null) return null;
-		return OafDecoder.decode(body).getEntity().getPerson().getMetadata().getFullname().getValue();
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupPersonBean.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupPersonBean.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupPersonBean.java	(nonexistent)
@@ -1,63 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.Serializable;
-import java.util.Set;
-
-import org.apache.hadoop.io.Text;
-
-import com.google.gson.Gson;
-
-public class DedupPersonBean implements Serializable {
-
-	/**
-	 * 
-	 */
-	private static final long serialVersionUID = -4107429095178989581L;
-
-	private String id;
-	private String name;
-	private Set<String> results;
-
-	public DedupPersonBean() {
-	}
-
-	public DedupPersonBean(String id, String name, Set<String> results) {
-		this.id = id;
-		this.name = name;
-		this.results = results;
-	}
-
-	public String getId() {
-		return id;
-	}
-
-	public void setId(String id) {
-		this.id = id;
-	}
-
-	public String getName() {
-		return name;
-	}
-
-	public void setName(String name) {
-		this.name = name;
-	}
-
-	public Set<String> getResults() {
-		return results;
-	}
-
-	public void setResults(Set<String> results) {
-		this.results = results;
-	}
-
-	public Text toText() {
-		return new Text((new Gson()).toJson(this));
-	}
-
-	public static DedupPersonBean fromText(Text text) {
-		return (new Gson()).fromJson(text.toString(), DedupPersonBean.class);
-
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/RootEntity.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/RootEntity.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/RootEntity.java	(nonexistent)
@@ -1,44 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.util.List;
-
-import com.google.gson.Gson;
-
-public class RootEntity {
-
-	private String joaf;
-	private List<String> mergedIds;
-
-	public RootEntity() {}
-
-	public RootEntity(final String joaf, final List<String> mergedIds) {
-		super();
-		this.setJoaf(joaf);
-		this.setMergedIds(mergedIds);
-	}
-
-	public static RootEntity decode(final String s) {
-		return new Gson().fromJson(s, RootEntity.class);
-	}
-
-	@Override
-	public String toString() {
-		return new Gson().toJson(this);
-	}
-
-	public String getJoaf() {
-		return joaf;
-	}
-
-	public void setJoaf(final String joaf) {
-		this.joaf = joaf;
-	}
-
-	public List<String> getMergedIds() {
-		return mergedIds;
-	}
-
-	public void setMergedIds(final List<String> mergedIds) {
-		this.mergedIds = mergedIds;
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupRootsToCsvReducer.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupRootsToCsvReducer.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupRootsToCsvReducer.java	(nonexistent)
@@ -1,86 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.nio.charset.Charset;
-
-import org.apache.commons.lang.StringUtils;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Reducer;
-import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;
-
-import com.google.common.base.Function;
-import com.google.common.base.Joiner;
-import com.google.common.collect.Iterables;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.pace.config.DedupConfig;
-
-public class DedupRootsToCsvReducer extends Reducer<ImmutableBytesWritable, ImmutableBytesWritable, Text, Text> {
-
-	/**
-	 * logger.
-	 */
-	private static final Log log = LogFactory.getLog(DedupRootsToCsvReducer.class); // NOPMD by marko on 11/24/08 5:02 PM
-
-	private static final String COUNTER_GROUP = "csv";
-
-	enum Tables {
-		Groups, NativeGroups, NativeEntities
-	}
-
-	private DedupConfig dedupConf;
-
-	private String DELIM;
-
-	private String WRAP;
-
-	private Text tKey;
-	private Text tValue;
-	private MultipleOutputs<Text, Text> mos;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		super.setup(context);
-		tKey = new Text();
-		tValue = new Text();
-
-		mos = new MultipleOutputs<Text, Text>(context);
-		dedupConf = DedupConfig.load(context.getConfiguration().get(JobParams.DEDUP_CONF));
-
-		log.info("wf conf: " + dedupConf.toString());
-
-		DELIM = context.getConfiguration().get("mapred.textoutputformat.separator", "!");
-		WRAP = context.getConfiguration().get("mapred.textoutputformat.wrapper", "#");
-
-		log.info("using field DELIMITER: '" + DELIM + "'");
-	}
-
-	@Override
-	protected void reduce(final ImmutableBytesWritable key, final Iterable<ImmutableBytesWritable> values, final Context context) throws IOException,
-	InterruptedException {
-
-		final String csv = Joiner.on(DELIM).join(Iterables.transform(values, new Function<ImmutableBytesWritable, String>() {
-
-			@Override
-			public String apply(final ImmutableBytesWritable ibw) {
-
-				return new String(ibw.copyBytes(), Charset.forName("UTF-8"));
-			}
-		}));
-
-		tKey.set(StringUtils.substringBefore(csv, DELIM));
-		tValue.set(StringUtils.substringAfter(csv, DELIM));
-
-		mos.write(Tables.Groups.toString(), tKey, tValue, Tables.Groups.toString());
-		context.getCounter(COUNTER_GROUP, "groups").increment(StringUtils.countMatches(csv, DELIM));
-	}
-
-	@Override
-	public void cleanup(final Context context) throws IOException, InterruptedException {
-		mos.close();
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupBuildRootsReducer.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupBuildRootsReducer.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupBuildRootsReducer.java	(nonexistent)
@@ -1,218 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.util.List;
-
-import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableReducer;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import com.google.protobuf.InvalidProtocolBufferException;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.hbase.index.config.RelClasses;
-import eu.dnetlib.data.mapreduce.util.DedupUtils;
-import eu.dnetlib.data.mapreduce.util.OafDecoder;
-import eu.dnetlib.data.mapreduce.util.OafHbaseUtils;
-import eu.dnetlib.data.mapreduce.util.OafRelDecoder;
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.proto.RelTypeProtos.RelType;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.data.transform.OafEntityMerger;
-import eu.dnetlib.pace.config.DedupConfig;
-
-public class DedupBuildRootsReducer extends TableReducer<Text, ImmutableBytesWritable, ImmutableBytesWritable> {
-
-	private enum OafPatch {
-		rootToEntity, entityToRoot
-	}
-
-	private static final boolean WRITE_TO_WAL = false;
-
-	private DedupConfig dedupConf;
-
-	private RelClasses relClasses;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		super.setup(context);
-		dedupConf = DedupConfig.load(context.getConfiguration().get(JobParams.DEDUP_CONF));
-		System.out.println("dedup buildRoots reducer\n\nwf conf: " + dedupConf.toString());
-
-		final String relClassJson = context.getConfiguration().get("relClasses");
-		System.out.println("relClassesJson:\n" + relClassJson);
-		relClasses = RelClasses.fromJSon(relClassJson);
-		System.out.println("relClasses:\n" + relClasses);
-	}
-
-	@Override
-	protected void reduce(final Text key, final Iterable<ImmutableBytesWritable> values, final Context context) throws IOException, InterruptedException {
-
-		// ensures we're dealing with a root, otherwise returns
-		if (!DedupUtils.isRoot(key.toString())) {
-			System.err.println("aborting DedupBuildRootsReducer, found non-root key: " + key);
-			context.getCounter("DedupBuildRootsReducer", "aborted").increment(1);
-			return;
-		}
-
-		final byte[] rowkey = Bytes.toBytes(key.toString());
-		final List<Oaf> entities = Lists.newArrayList();
-
-		for (final Oaf oaf : toOaf(values)) {
-			switch (oaf.getKind()) {
-			case entity:
-				entities.add(oaf);
-				break;
-			case relation:
-				handleRels(context, rowkey, oaf, true);
-				break;
-			default:
-				break;
-			}
-		}
-
-		// build and emit the root body
-		final Oaf.Builder builder = OafEntityMerger.merge(dedupConf, key.toString(), entities);
-		if (entities.size() < JobParams.MAX_COUNTERS) {
-			context.getCounter(dedupConf.getWf().getEntityType() + " root group size", lpad(entities.size())).increment(1);
-		} else {
-			context.getCounter(dedupConf.getWf().getEntityType() + " root group size", "> " + JobParams.MAX_COUNTERS).increment(1);
-		}
-
-		emit(context, rowkey, dedupConf.getWf().getEntityType(), DedupUtils.BODY_S, builder.build().toByteArray(), "root");
-
-		// add person rels TODO: remove this hack
-		// context.getCounter("hack", "personResult out").increment(personMap.size());
-
-	}
-
-	private Iterable<Oaf> toOaf(final Iterable<ImmutableBytesWritable> values) {
-		return Iterables.transform(values, OafHbaseUtils.oafDecoder());
-	}
-
-	private void handleRels(final Context context, final byte[] rowkey, final Oaf oaf, final boolean hack) throws IOException, InterruptedException,
-	InvalidProtocolBufferException {
-
-		if (hack && checkHack(new String(rowkey), oaf)) {
-			context.getCounter("hack", "personResult in").increment(1);
-		} else {
-
-			// emit relation from the root to the related entities
-			OafDecoder decoder = rootToEntity(rowkey, oaf);
-			emit(context, rowkey, decoder.getCFQ(), decoder.relTargetId(), decoder.toByteArray(), "[root -> entity]");
-
-			// emit relation from the related entities to the root
-			decoder = entityToRoot(rowkey, oaf);
-			final byte[] revKey = Bytes.toBytes(decoder.relSourceId());
-			emit(context, revKey, decoder.getCFQ(), new String(rowkey), decoder.toByteArray(), "[entity -> root]");
-		}
-		// mark relation from the related entities to the duplicate as deleted
-		OafDecoder decoder = markDeleted(oaf, true);
-		byte[] revKey = Bytes.toBytes(decoder.relSourceId());
-		emit(context, revKey, decoder.getCFQ(), decoder.relTargetId(), decoder.toByteArray(), "mark deleted [dup -> entity]");
-
-		// mark relation from the related entities to the duplicate as deleted
-		decoder = markDeleted(oaf, false);
-		revKey = Bytes.toBytes(decoder.relSourceId());
-		emit(context, revKey, decoder.getCFQ(), decoder.relTargetId(), decoder.toByteArray(), "mark deleted [entity -> dup]");
-	}
-
-	private void emit(final Context context, final byte[] rowkey, final String family, final String qualifier, final byte[] value, final String label)
-			throws IOException, InterruptedException {
-		final Put put = new Put(rowkey).add(Bytes.toBytes(family), Bytes.toBytes(qualifier), value);
-		put.setWriteToWAL(WRITE_TO_WAL);
-		context.write(new ImmutableBytesWritable(rowkey), put);
-		context.getCounter(family, label).increment(1);
-	}
-
-	// /////////////////
-
-	public boolean checkHack(final String root, final Oaf oaf) {
-
-		boolean res;
-		if (dedupConf.getWf().getEntityType().equals(Type.result.toString()) && oaf.getRel().getRelType().toString().equals(RelType.personResult.toString())
-				&& !md5matches(root, oaf.getRel().getSource())) {
-
-			res = true;
-		} else {
-			res = false;
-		}
-
-		// if (root.equals("50|dedup_wf_001::92f6197ea6f16ae554755aced832fb6f")) {
-		// System.out.println("##################");
-		// System.out.println("root  : " + root);
-		// System.out.println("source: " + oaf.getRel().getSource());
-		// System.out.println("ckeck:  " + res);
-		// }
-
-		return res;
-	}
-
-	private boolean md5matches(final String id1, final String id2) {
-		return id1.replaceAll("^.*\\:\\:", "").equals(id2.replaceAll("^.*\\:\\:", ""));
-	}
-
-	private OafDecoder rootToEntity(final byte[] rootRowkey, final Oaf rel) {
-		return patchRelations(rootRowkey, rel, OafPatch.rootToEntity);
-	}
-
-	private OafDecoder entityToRoot(final byte[] rootRowkey, final Oaf rel) {
-		return patchRelations(rootRowkey, rel, OafPatch.entityToRoot);
-	}
-
-	private OafDecoder markDeleted(final Oaf rel, final boolean reverse) {
-		return deleteRelations(rel, reverse);
-	}
-
-	// patches relation objects setting the source field with the root id
-	private OafDecoder patchRelations(final byte[] rootRowkey, final Oaf rel, final OafPatch patchKind) {
-		final OafRelDecoder decoder = OafRelDecoder.decode(rel.getRel());
-		final Oaf.Builder builder = Oaf.newBuilder(rel);
-		builder.getDataInfoBuilder().setInferred(true).setDeletedbyinference(false);
-		switch (patchKind) {
-		case rootToEntity:
-			// builder.getDataInfoBuilder().setInferenceprovenance("dedup (BuildRoots p:rootToEntity)");
-			builder.getRelBuilder().setSource(new String(rootRowkey));
-			break;
-
-		case entityToRoot:
-			builder.setRel(decoder.setClassId(relClasses.getInverse(rel.getRel().getRelClass())));
-			// builder.getDataInfoBuilder().setInferenceprovenance("dedup (BuildRoots p:entityToRoot)");
-			builder.getRelBuilder().setSource(builder.getRel().getTarget());
-			builder.getRelBuilder().setTarget(new String(rootRowkey));
-			break;
-
-		default:
-			break;
-		}
-
-		return OafDecoder.decode(builder.build());
-	}
-
-	private OafDecoder deleteRelations(final Oaf rel, final boolean reverse) {
-		final Oaf.Builder builder = Oaf.newBuilder(rel);
-		// builder.getDataInfoBuilder().setInferenceprovenance("dedup (BuildRoots d: " + reverse + ")");
-		builder.getDataInfoBuilder().setDeletedbyinference(true);
-
-		if (reverse) {
-			final OafRelDecoder decoder = OafRelDecoder.decode(rel.getRel());
-			builder.setRel(decoder.setClassId(relClasses.getInverse(rel.getRel().getRelClass())));
-			// swap source and target
-			final String tmp = builder.getRel().getSource();
-			builder.getRelBuilder().setSource(builder.getRel().getTarget());
-			builder.getRelBuilder().setTarget(tmp);
-		}
-
-		return OafDecoder.decode(builder.build());
-	}
-
-	private String lpad(final int s) {
-		return StringUtils.leftPad(String.valueOf(s), 5);
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/FindDedupCandidatePersonsReducer.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/FindDedupCandidatePersonsReducer.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/FindDedupCandidatePersonsReducer.java	(nonexistent)
@@ -1,115 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.util.HashSet;
-import java.util.LinkedList;
-import java.util.Map;
-import java.util.Queue;
-import java.util.Set;
-
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableReducer;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-
-import com.google.common.collect.Maps;
-import com.google.common.collect.Sets;
-
-import eu.dnetlib.data.mapreduce.hbase.HBaseTableUtils.VolatileColumnFamily;
-import eu.dnetlib.pace.model.PersonComparatorUtils;
-
-public class FindDedupCandidatePersonsReducer extends TableReducer<Text, Text, ImmutableBytesWritable> {
-
-	private static final boolean WRITE_TO_WAL = false;
-	private static final int LIMIT = 5000;
-
-	@Override
-	protected void setup(Context context) throws IOException, InterruptedException {
-
-	}
-
-	@Override
-	protected void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
-		System.out.println("\nReducing key: " + key);
-
-		Set<String> candidates = Sets.newHashSet();
-		Map<String, Set<String>> resultIds = Maps.newHashMap();
-
-		Queue<DedupPersonBean> queue = prepare(context, key, values);
-		while (!queue.isEmpty()) {
-
-			final DedupPersonBean pivot = queue.remove();
-
-			for (DedupPersonBean curr : queue) {
-
-				if (PersonComparatorUtils.areSimilar(pivot.getName(), curr.getName())) {
-					System.out.println("- Similar persons: [" + pivot.getName() + "] - [" + curr.getName() + "]");
-					candidates.add(pivot.getId());
-					candidates.add(curr.getId());
-
-					collectResultIds(resultIds, pivot);
-					collectResultIds(resultIds, curr);
-				}
-			}
-		}
-
-		emitCandidates(context, candidates);
-		emitResultCandidates(context, resultIds);
-	}
-
-	private void collectResultIds(Map<String, Set<String>> resultIds, DedupPersonBean person) {
-		if (!resultIds.containsKey(person.getId())) {
-			resultIds.put(person.getId(), new HashSet<String>());
-		}
-		resultIds.get(person.getId()).addAll(person.getResults());
-	}
-
-	private Queue<DedupPersonBean> prepare(Context context, Text key, Iterable<Text> values) {
-		final Queue<DedupPersonBean> queue = new LinkedList<DedupPersonBean>();
-
-		for (Text i : values) {
-			queue.add(DedupPersonBean.fromText(i));
-
-			if (queue.size() > LIMIT) {
-				context.getCounter("Comparison list > " + LIMIT, "'" + key.toString() + "', --> " + context.getTaskAttemptID()).increment(1);
-				System.out.println("breaking out after limit (" + LIMIT + ") for key '" + key);
-				break;
-			}
-		}
-
-		return queue;
-	}
-
-	private void emitCandidates(Context context, Set<String> candidates) throws IOException, InterruptedException {
-		byte[] cf = Bytes.toBytes(VolatileColumnFamily.dedup.toString());
-		byte[] col = Bytes.toBytes("isCandidate");
-		byte[] val = Bytes.toBytes("");
-
-		for (String s : candidates) {
-			byte[] id = Bytes.toBytes(s);
-			Put put = new Put(id).add(cf, col, val);
-			put.setWriteToWAL(WRITE_TO_WAL);
-			context.write(new ImmutableBytesWritable(id), put);
-		}
-		context.getCounter(getClass().getSimpleName(), "N. Put. (persons)").increment(candidates.size());
-	}
-
-	private void emitResultCandidates(Context context, Map<String, Set<String>> resultIds) throws IOException, InterruptedException {
-		byte[] cf = Bytes.toBytes(VolatileColumnFamily.dedupPerson.toString());
-		byte[] val = Bytes.toBytes("");
-
-		for (String personId : resultIds.keySet()) {
-			byte[] col = Bytes.toBytes(personId);
-
-			for (String s : resultIds.get(personId)) {
-				byte[] id = Bytes.toBytes(s);
-				Put put = new Put(id).add(cf, col, val);
-				put.setWriteToWAL(WRITE_TO_WAL);
-				context.write(new ImmutableBytesWritable(id), put);
-			}
-			context.getCounter(getClass().getSimpleName(), "N. Put. (results)").increment(resultIds.get(personId).size());
-		}
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupPersonMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupPersonMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupPersonMapper.java	(nonexistent)
@@ -1,73 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.io.Text;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.util.DedupUtils;
-import eu.dnetlib.data.mapreduce.util.OafDecoder;
-import eu.dnetlib.pace.config.DedupConfig;
-import eu.dnetlib.pace.model.Person;
-
-public class DedupPersonMapper extends TableMapper<Text, ImmutableBytesWritable> {
-
-	private DedupConfig dedupConf;
-
-	private Text rowKey;
-
-	private ImmutableBytesWritable ibw;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		dedupConf = DedupConfig.load(context.getConfiguration().get(JobParams.DEDUP_CONF));
-		rowKey = new Text();
-		ibw = new ImmutableBytesWritable();
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable keyIn, final Result result, final Context context) throws IOException, InterruptedException {
-		// System.out.println("got key: " + new String(keyIn.copyBytes()));
-
-		if (DedupUtils.isRoot(new String(keyIn.copyBytes()))) {
-			context.getCounter(dedupConf.getWf().getEntityType(), "roots skipped").increment(1);
-			return;
-		}
-		final byte[] body = result.getValue(dedupConf.getWf().getEntityType().getBytes(), DedupUtils.BODY_B);
-
-		if (body != null) {
-			try {
-				final OafDecoder decoder = OafDecoder.decode(body);
-
-				final String hash = new Person(decoder.getEntity().getPerson().getMetadata().getFullname().getValue(), false).hash();
-				// String hash = new Person(getPersonName(decoder), true).hash();
-
-				rowKey.set(hash);
-				ibw.set(body);
-				context.write(rowKey, ibw);
-
-			} catch (final Throwable e) {
-				System.out.println("GOT EX " + e);
-				e.printStackTrace(System.err);
-				context.getCounter(dedupConf.getWf().getEntityType(), e.getClass().toString()).increment(1);
-			}
-		} else {
-			context.getCounter(dedupConf.getWf().getEntityType(), "missing body").increment(1);
-		}
-	}
-
-	// private String getPersonName(OafDecoder decoder) {
-	// Metadata m = decoder.getEntity().getPerson().getMetadata();
-	// String secondnames = Joiner.on(" ").join(m.getSecondnamesList());
-	//
-	// return isValid(m.getFullname()) ? m.getFullname() : (secondnames + ", " + m.getFirstname());
-	// }
-
-	// private boolean isValid(String fullname) {
-	// return fullname != null && !fullname.isEmpty();
-	// }
-
-}

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupPersonMapper.java
___________________________________________________________________
Deleted: svn:mime-type
## -1 +0,0 ##
-text/plain
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupFindRootsMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupFindRootsMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupFindRootsMapper.java	(nonexistent)
@@ -1,112 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.util.Map;
-
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-
-import com.google.protobuf.InvalidProtocolBufferException;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.util.DedupUtils;
-import eu.dnetlib.data.proto.DedupProtos.Dedup;
-import eu.dnetlib.data.proto.KindProtos.Kind;
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.proto.OafProtos.OafRel.Builder;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.data.transform.xml.AbstractDNetOafXsltFunctions;
-import eu.dnetlib.pace.config.DedupConfig;
-
-public class DedupFindRootsMapper extends TableMapper<ImmutableBytesWritable, Put> {
-
-	private static final boolean WRITE_TO_WAL = false;
-
-	private DedupConfig dedupConf;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		dedupConf = DedupConfig.load(context.getConfiguration().get(JobParams.DEDUP_CONF));
-		System.out.println("dedup findRoots mapper\nwf conf: " + dedupConf.toString());
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable rowkey, final Result value, final Context context) throws IOException, InterruptedException {
-		// System.out.println("Find root mapping: " + new String(rowkey.copyBytes()));
-
-		final Type type = Type.valueOf(dedupConf.getWf().getEntityType());
-		final Map<byte[], byte[]> similarRels = value.getFamilyMap(DedupUtils.getSimilarityCFBytes(type));
-
-		if ((similarRels != null) && !similarRels.isEmpty()) {
-			final ByteBuffer min = findMin(ByteBuffer.wrap(rowkey.get()), similarRels.keySet());
-
-			final byte[] row = rowkey.copyBytes();
-			final byte[] root = DedupUtils.newIdBytes(min, dedupConf.getWf().getDedupRun());
-
-			// System.out.println("Found root: " + new String(root));
-
-			emitDedupRel(context, DedupUtils.getDedupCF_mergedInBytes(type), row, root, buildRel(row, root, Dedup.RelName.isMergedIn));
-			emitDedupRel(context, DedupUtils.getDedupCF_mergesBytes(type), root, row, buildRel(root, row, Dedup.RelName.merges));
-
-			context.getCounter(dedupConf.getWf().getEntityType(), "dedupRel (x2)").increment(1);
-
-			// marks the original body deleted
-			emitBody(context, row, value.getValue(Bytes.toBytes(dedupConf.getWf().getEntityType()), DedupUtils.BODY_B));
-
-		} else {
-			context.getCounter(dedupConf.getWf().getEntityType(), "row not in similarity mesh").increment(1);
-		}
-	}
-
-	private ByteBuffer findMin(ByteBuffer min, final Iterable<byte[]> keys) {
-		for (final byte[] q : keys) {
-			final ByteBuffer iq = ByteBuffer.wrap(q);
-			if (min.compareTo(iq) > 0) {
-				min = iq;
-			}
-		}
-		return min;
-	}
-
-	private void emitBody(final Context context, final byte[] row, final byte[] body) throws InvalidProtocolBufferException, IOException, InterruptedException {
-		if (body == null) {
-			context.getCounter(dedupConf.getWf().getEntityType(), "missing body").increment(1);
-			System.err.println("missing body: " + new String(row));
-			return;
-		}
-		final Oaf prototype = Oaf.parseFrom(body);
-
-		if (prototype.getDataInfo().getDeletedbyinference()) {
-			context.getCounter(dedupConf.getWf().getEntityType(), "bodies already deleted").increment(1);
-		} else {
-			final Oaf.Builder oafRoot = Oaf.newBuilder(prototype);
-			oafRoot.getDataInfoBuilder().setDeletedbyinference(true).setInferred(true).setInferenceprovenance("dedup");
-			final byte[] family = Bytes.toBytes(dedupConf.getWf().getEntityType());
-			final Put put = new Put(row).add(family, DedupUtils.BODY_B, oafRoot.build().toByteArray());
-			put.setWriteToWAL(WRITE_TO_WAL);
-			context.write(new ImmutableBytesWritable(row), put);
-			context.getCounter(dedupConf.getWf().getEntityType(), "bodies marked deleted").increment(1);
-		}
-	}
-
-	private byte[] buildRel(final byte[] from, final byte[] to, final Dedup.RelName relClass) {
-		final Builder oafRel = DedupUtils.getDedup(dedupConf, new String(from), new String(to), relClass);
-		final Oaf oaf =
-				Oaf.newBuilder().setKind(Kind.relation).setTimestamp(System.currentTimeMillis())
-				.setDataInfo(AbstractDNetOafXsltFunctions.getDataInfo(null, "", "0.8", false, true).setInferenceprovenance("dedup")).setRel(oafRel)
-						.build();
-		return oaf.toByteArray();
-	}
-
-	private void emitDedupRel(final Context context, final byte[] cf, final byte[] from, final byte[] to, final byte[] value) throws IOException,
-	InterruptedException {
-		final Put put = new Put(from).add(cf, to, value);
-		put.setWriteToWAL(WRITE_TO_WAL);
-		context.write(new ImmutableBytesWritable(from), put);
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupSimilarityToActionsMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupSimilarityToActionsMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupSimilarityToActionsMapper.java	(nonexistent)
@@ -1,77 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import org.apache.commons.collections.MapUtils;
-import org.apache.commons.lang.StringUtils;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-
-import eu.dnetlib.actionmanager.actions.ActionFactory;
-import eu.dnetlib.actionmanager.actions.AtomicAction;
-import eu.dnetlib.actionmanager.common.Agent;
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.pace.config.DedupConfig;
-
-public class DedupSimilarityToActionsMapper extends TableMapper<ImmutableBytesWritable, Put> {
-
-	private static final Log log = LogFactory.getLog(DedupSimilarityToActionsMapper.class); // NOPMD by marko on 11/24/08 5:02 PM
-
-	private static final String RAW_SET = "rawSetId";
-	private static final String SIMILARITY_CF = "similarityCF";
-
-	private String rawSet = null;
-
-	private String similarityCF = null;
-
-	private DedupConfig dedupConf = null;
-
-	private ActionFactory actionFactory = null;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		rawSet = context.getConfiguration().get(RAW_SET);
-		if (StringUtils.isBlank(rawSet)) throw new IOException("Input parameter (" + RAW_SET + ") is missing or empty: '" + rawSet + "'");
-		log.info("raw set: '" + rawSet + "'");
-
-		similarityCF = context.getConfiguration().get(SIMILARITY_CF);
-		if (StringUtils.isBlank(similarityCF)) throw new IOException("Input parameter (" + SIMILARITY_CF + ") is missing or empty: '" + similarityCF + "'");
-		log.info("similarityCF: '" + similarityCF + "'");
-
-		dedupConf = DedupConfig.load(context.getConfiguration().get(JobParams.DEDUP_CONF));
-
-		log.info("wf conf: " + dedupConf.toString());
-
-		actionFactory = new ActionFactory();
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable key, final Result value, final Context context) throws IOException, InterruptedException {
-
-		final Map<byte[], byte[]> sMap = value.getFamilyMap(Bytes.toBytes(similarityCF));
-		if (MapUtils.isEmpty(sMap)) return;
-
-		final Agent agent = new Agent("dedup", "Deduplication", Agent.AGENT_TYPE.algo);
-
-		for (final Entry<byte[], byte[]> similarity : sMap.entrySet()) {
-
-			final String targetKey = new String(key.copyBytes());
-			final String qualifier = new String(similarity.getKey());
-			final AtomicAction aa = actionFactory.createAtomicAction(rawSet, agent, targetKey, similarityCF, qualifier, null);
-
-			for (final Put put : aa.asPutOperations(null, null, null, null)) {
-
-				context.write(new ImmutableBytesWritable(Bytes.toBytes(aa.getRowKey())), put);
-				context.getCounter(dedupConf.getWf().getEntityType(), "similarity2actions").increment(1);
-			}
-		}
-
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupPersonReducer.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupPersonReducer.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupPersonReducer.java	(nonexistent)
@@ -1,147 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.util.Queue;
-
-import org.apache.commons.lang.StringUtils;
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableReducer;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import com.google.protobuf.InvalidProtocolBufferException;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.util.DedupUtils;
-import eu.dnetlib.data.mapreduce.util.OafDecoder;
-import eu.dnetlib.data.mapreduce.util.OafHbaseUtils;
-import eu.dnetlib.data.mapreduce.util.OafRowKeyDecoder;
-import eu.dnetlib.data.proto.DedupProtos.Dedup;
-import eu.dnetlib.data.proto.KindProtos.Kind;
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.proto.OafProtos.OafRel;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.data.transform.xml.AbstractDNetOafXsltFunctions;
-import eu.dnetlib.pace.config.DedupConfig;
-
-public class DedupPersonReducer extends TableReducer<Text, ImmutableBytesWritable, ImmutableBytesWritable> {
-
-	private static final boolean WRITE_TO_WAL = false;
-
-	private static final int MAX_Q_SIZE = 3000;
-
-	private DedupConfig dedupConf;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		dedupConf = DedupConfig.load(context.getConfiguration().get(JobParams.DEDUP_CONF));
-	}
-
-	@Override
-	protected void reduce(final Text key, final Iterable<ImmutableBytesWritable> values, final Context context) throws IOException, InterruptedException {
-		try {
-			final Queue<OafDecoder> q = prepare(key, values, context);
-			if (q.size() > 1) {
-
-				if (q.size() < JobParams.MAX_COUNTERS) {
-					context.getCounter(dedupConf.getWf().getEntityType() + " root group size", lpad(q.size())).increment(1);
-				} else {
-					context.getCounter(dedupConf.getWf().getEntityType() + " root group size", "> " + JobParams.MAX_COUNTERS).increment(1);
-				}
-				final String min = findMin(Iterables.transform(q, eu.dnetlib.data.transform.OafUtils.idDecoder()));
-				if (min == null) {
-					context.getCounter(dedupConf.getWf().getEntityType(), "unable to find min").increment(1);
-					return;
-				}
-				final String rootId = DedupUtils.newId(min, dedupConf.getWf().getDedupRun());
-
-				while (!q.isEmpty()) {
-					markDuplicate(context, rootId, q.remove());
-				}
-			} else {
-				context.getCounter(dedupConf.getWf().getEntityType(), "1").increment(1);
-			}
-		} catch (final Throwable e) {
-			System.out.println("GOT EX " + e);
-			e.printStackTrace(System.err);
-			context.getCounter(dedupConf.getWf().getEntityType(), e.getClass().toString()).increment(1);
-		}
-	}
-
-	private Queue<OafDecoder> prepare(final Text key, final Iterable<ImmutableBytesWritable> values, final Context context) {
-		final Queue<OafDecoder> q = Lists.newLinkedList();
-		for (final OafDecoder decoder : Iterables.transform(values, OafHbaseUtils.decoder())) {
-			q.add(decoder);
-			if (q.size() >= MAX_Q_SIZE) {
-				context.getCounter("[" + key.toString() + "]", "size > " + MAX_Q_SIZE).increment(1);
-				break;
-			}
-		}
-		return q;
-	}
-
-	public static String findMin(final Iterable<String> keys) {
-		String min = Iterables.getFirst(keys, null);
-		for (final String iq : keys) {
-			if (min.compareTo(iq) > 0) {
-				min = iq;
-			}
-		}
-		return min;
-	}
-
-	private void markDuplicate(final Context context, final String rootId, final OafDecoder decoder) throws InvalidProtocolBufferException, IOException,
-			InterruptedException {
-
-		final Oaf.Builder builder = Oaf.newBuilder(decoder.getOaf());
-		builder.getDataInfoBuilder().setDeletedbyinference(true).setInferenceprovenance("dedup person");
-
-		final Oaf oaf = builder.build();
-		final byte[] oafId = Bytes.toBytes(oaf.getEntity().getId());
-
-		// writes the body, marked as deleted
-		final String entityName = dedupConf.getWf().getEntityType();
-		emit(context, oafId, entityName, DedupUtils.BODY_B, oaf.toByteArray());
-		context.getCounter(entityName, "marked as deleted").increment(1);
-
-		// writes the dedupRels in both directions
-		final Type entityType = Type.valueOf(entityName);
-		final byte[] rowkey = Bytes.toBytes(rootId);
-
-		final String merges = DedupUtils.getDedupCF_merges(entityType);
-		emit(context, rowkey, merges, oafId, buildRel(rowkey, oafId, Dedup.RelName.merges));
-
-		final String mergedIn = DedupUtils.getDedupCF_mergedIn(entityType);
-		emit(context, oafId, mergedIn, rowkey, buildRel(oafId, rowkey, Dedup.RelName.isMergedIn));
-
-		context.getCounter(entityName, merges).increment(1);
-		context.getCounter(entityName, mergedIn).increment(1);
-	}
-
-	private void emit(final Context context, final byte[] rowkey, final String family, final byte[] qualifier, final byte[] value) throws IOException,
-			InterruptedException {
-
-		final Put put = new Put(OafRowKeyDecoder.decode(rowkey).getKey().getBytes());
-		put.setWriteToWAL(WRITE_TO_WAL);
-		put.add(Bytes.toBytes(family), qualifier, value);
-
-		context.write(new ImmutableBytesWritable(rowkey), put);
-	}
-
-	private byte[] buildRel(final byte[] from, final byte[] to, final Dedup.RelName relClass) {
-		final OafRel.Builder oafRel = DedupUtils.getDedup(dedupConf, new String(from), new String(to), relClass);
-		final Oaf oaf =
-				Oaf.newBuilder().setKind(Kind.relation).setTimestamp(System.currentTimeMillis())
-						.setDataInfo(AbstractDNetOafXsltFunctions.getDataInfo(null, "", "0.8", false, true).setInferenceprovenance("dedup")).setRel(oafRel)
-				.build();
-		return oaf.toByteArray();
-	}
-
-	private String lpad(final int s) {
-		return StringUtils.leftPad(String.valueOf(s), String.valueOf(MAX_Q_SIZE).length());
-	}
-
-}

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupPersonReducer.java
___________________________________________________________________
Deleted: svn:mime-type
## -1 +0,0 ##
-text/plain
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/FindPersonCoauthorsMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/FindPersonCoauthorsMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/FindPersonCoauthorsMapper.java	(nonexistent)
@@ -1,66 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.NavigableMap;
-
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-import org.apache.http.HttpResponse;
-import org.apache.http.client.HttpClient;
-import org.apache.http.client.methods.HttpPost;
-import org.apache.http.client.methods.HttpUriRequest;
-import org.apache.http.impl.client.DefaultHttpClient;
-import org.apache.http.params.BasicHttpParams;
-import org.apache.http.params.HttpParams;
-
-import com.google.common.collect.Lists;
-import com.google.gson.Gson;
-
-import eu.dnetlib.data.mapreduce.hbase.HBaseTableUtils.VolatileColumnFamily;
-import eu.dnetlib.data.proto.RelTypeProtos.RelType;
-
-public class FindPersonCoauthorsMapper extends TableMapper<Text, Text> {
-	private final HttpClient client = new DefaultHttpClient();
-
-	private final String url = "http://146.48.87.97:8888/addData";
-
-	@Override
-	protected void setup(Context context) {
-		//		url = context.getConfiguration().get("dedup.person.coauthors.service.url");
-	}
-
-	@Override
-	protected void map(ImmutableBytesWritable rowkey, Result row, Context context) throws IOException, InterruptedException {
-		NavigableMap<byte[], byte[]> candidates = row.getFamilyMap(Bytes.toBytes(VolatileColumnFamily.dedupPerson.toString()));
-		if (candidates == null || candidates.isEmpty()) {
-			return;
-		}
-
-		List<String> coauthors = Lists.newArrayList();
-		for (byte[] b : row.getFamilyMap(Bytes.toBytes(RelType.personResult.toString())).keySet()) {
-			coauthors.add(Bytes.toString(b));
-		}
-
-		for (byte[] candidate : candidates.keySet()) {
-			emit(context, Bytes.toString(candidate), coauthors);
-		}
-	}
-
-	private void emit(Context context, String candidate, List<String> coauthors) {
-		try {
-			HttpUriRequest request = new HttpPost(url);
-			HttpParams params = new BasicHttpParams();
-			params.setParameter("id", candidate);
-			params.setParameter("data", (new Gson().toJson(coauthors)));
-			request.setParams(params);
-			HttpResponse response = client.execute(request);
-			context.getCounter("HTTP call", "code " + response.getStatusLine().getStatusCode()).increment(1);
-		} catch (Exception e) {
-			context.getCounter("HTTP call", "Exception " + e.getClass()).increment(1);
-		}
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dedup/DedupMapper.java	(nonexistent)
@@ -1,98 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dedup;
-
-import java.io.IOException;
-import java.util.Collection;
-import java.util.List;
-import java.util.Map;
-
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-
-import com.google.common.collect.Maps;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.util.DedupUtils;
-import eu.dnetlib.data.mapreduce.util.OafDecoder;
-import eu.dnetlib.data.proto.OafProtos.OafEntity;
-import eu.dnetlib.data.proto.TypeProtos.Type;
-import eu.dnetlib.pace.clustering.BlacklistAwareClusteringCombiner;
-import eu.dnetlib.pace.config.DedupConfig;
-import eu.dnetlib.pace.model.MapDocument;
-import eu.dnetlib.pace.model.ProtoDocumentBuilder;
-
-public class DedupMapper extends TableMapper<Text, ImmutableBytesWritable> {
-
-	private static final Log log = LogFactory.getLog(DedupMapper.class);
-
-	private DedupConfig dedupConf;
-
-	private Map<String, List<String>> blackListMap = Maps.newHashMap();
-
-	private Text outKey;
-
-	private ImmutableBytesWritable ibw;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-
-		final String dedupConfJson = context.getConfiguration().get(JobParams.DEDUP_CONF);
-
-		log.info("pace conf strings");
-		log.info("pace conf: " + dedupConfJson);
-
-		dedupConf = DedupConfig.load(dedupConfJson);
-
-		blackListMap = dedupConf.getPace().getBlacklists();
-
-		outKey = new Text();
-		ibw = new ImmutableBytesWritable();
-
-		log.info("pace conf");
-		log.info("entity type: " + dedupConf.getWf().getEntityType());
-		log.info("clustering: " + dedupConf.getPace().getClustering());
-		log.info("conditions: " + dedupConf.getPace().getConditions());
-		log.info("fields: " + dedupConf.getPace().getModel());
-		log.info("blacklists: " + blackListMap);
-		log.info("wf conf: " + dedupConf.toString());
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable keyIn, final Result result, final Context context) throws IOException, InterruptedException {
-		// log.info("got key: " + new String(keyIn.copyBytes()));
-
-		final byte[] body = result.getValue(dedupConf.getWf().getEntityType().getBytes(), DedupUtils.BODY_B);
-
-		if (body != null) {
-
-			final OafEntity entity = OafDecoder.decode(body).getEntity();
-
-			context.getCounter(entity.getType().toString(), "decoded").increment(1);
-
-			if (entity.getType().equals(Type.valueOf(dedupConf.getWf().getEntityType()))) {
-
-				// TODO: remove this hack - here because we don't want to dedup datasets
-				if (entity.getType().equals(Type.result) && entity.getResult().getMetadata().getResulttype().getClassid().equals("dataset")) return;
-
-				// GeneratedMessage metadata = OafEntityDecoder.decode(entity).getEntity();
-				final MapDocument doc = ProtoDocumentBuilder.newInstance(Bytes.toString(keyIn.copyBytes()), entity, dedupConf.getPace().getModel());
-				emitNGrams(context, doc, BlacklistAwareClusteringCombiner.filterAndCombine(doc, dedupConf, blackListMap));
-			}
-		} else {
-			context.getCounter(dedupConf.getWf().getEntityType(), "missing body").increment(1);
-		}
-	}
-
-	private void emitNGrams(final Context context, final MapDocument doc, final Collection<String> ngrams) throws IOException, InterruptedException {
-		for (final String ngram : ngrams) {
-			outKey.set(ngram);
-			ibw.set(doc.toByteArray());
-			context.write(outKey, ibw);
-		}
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions2/PromoteActionsMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions2/PromoteActionsMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions2/PromoteActionsMapper.java	(nonexistent)
@@ -1,59 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.actions2;
-
-import java.io.IOException;
-import java.util.Set;
-
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-
-import com.google.common.base.Splitter;
-import com.google.common.collect.Sets;
-
-public class PromoteActionsMapper extends TableMapper<ImmutableBytesWritable, Put> {
-
-	private static final String LATEST_RAW_SETS = "latestRawSets";
-
-	private Set<byte[]> latestRawSets = Sets.newHashSet();
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-
-		final String s = context.getConfiguration().get(LATEST_RAW_SETS);
-		if (s != null) {
-			for (String set : Sets.newHashSet(Splitter.on(",").omitEmptyStrings().trimResults().split(s))) {
-				latestRawSets.add(Bytes.toBytes(set));
-			}
-		}
-
-		if (latestRawSets.isEmpty()) { throw new IOException("Input parameter (" + LATEST_RAW_SETS + ") is missing or empty: " + s); }
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable key, final Result value, final Context context) throws IOException, InterruptedException {
-		for (byte[] s1 : value.getFamilyMap(Bytes.toBytes("set")).keySet()) {
-			for (byte[] s2 : latestRawSets) {
-				if (Bytes.equals(s1, s2)) {
-					promoteAction(value, context);
-				}
-			}
-		}
-	}
-
-	private void promoteAction(final Result value, final Context context) throws IOException, InterruptedException {
-		final byte[] cf = Bytes.toBytes("target");
-		final byte[] tkey = value.getValue(cf, Bytes.toBytes("rowKey"));
-		final byte[] tcf = value.getValue(cf, Bytes.toBytes("columnFamily"));
-		final byte[] tc = value.getValue(cf, Bytes.toBytes("column"));
-		final byte[] tv = value.getValue(cf, Bytes.toBytes("content"));
-
-		final Put put = new Put(tkey);
-		put.add(tcf, tc, tv);
-
-		context.getCounter("Actions", Bytes.toString(tcf)).increment(1);
-
-		context.write(new ImmutableBytesWritable(tkey), put);
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions2/GarbageActionsMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions2/GarbageActionsMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/actions2/GarbageActionsMapper.java	(nonexistent)
@@ -1,73 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.actions2;
-
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.util.Set;
-
-import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-
-import com.google.common.base.Splitter;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Sets;
-
-import eu.dnetlib.miscutils.datetime.DateUtils;
-
-public class GarbageActionsMapper extends TableMapper<ImmutableBytesWritable, Delete> {
-
-	private static final String LATEST_RAW_SETS = "latestRawSets";
-
-	private static final String GARBAGE_TIME_MARING = "garbageTimeMargin";
-
-	private final Set<ByteBuffer> latestRawSets = Sets.newHashSet();
-
-	private long dateLimit;
-
-	private final long MAX_DATE_INTERVAL = 4 * 24 * 60 * 60 * 1000; // 4 days default value
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-
-		long garbageTimeMargin = MAX_DATE_INTERVAL;
-		try {
-			garbageTimeMargin = Long.parseLong(context.getConfiguration().get(GARBAGE_TIME_MARING));
-		} catch (NumberFormatException e) {}
-		this.dateLimit = DateUtils.now() - garbageTimeMargin;
-		System.out.println("dateLimit: " + dateLimit);
-
-		final String s = context.getConfiguration().get(LATEST_RAW_SETS);
-		if (s != null) {
-			for (String set : Sets.newHashSet(Splitter.on(",").omitEmptyStrings().trimResults().split(s))) {
-				latestRawSets.add(ByteBuffer.wrap(Bytes.toBytes(set)));
-			}
-		}
-
-		if (latestRawSets.isEmpty()) { throw new IOException("Input parameter (" + LATEST_RAW_SETS + ") is missing or empty: " + s); }
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable key, final Result value, final Context context) throws IOException, InterruptedException {
-		if (isInARecentRawSet(value)) { return; }
-
-		context.getCounter("Actions", "N. Deletes").increment(1);
-		context.write(key, new Delete(key.copyBytes()));
-	}
-
-	private boolean isInARecentRawSet(final Result value) {
-		for (byte[] s1 : value.getFamilyMap(Bytes.toBytes("set")).keySet()) {
-			if (isRecentRawSet(s1)) { return true; }
-		}
-		return false;
-	}
-
-	private boolean isRecentRawSet(final byte[] rawSet) {
-		final String date = Iterables.getLast(Splitter.on("_").split(Bytes.toString(rawSet)));
-
-		if (Long.parseLong(date) > this.dateLimit || latestRawSets.contains(ByteBuffer.wrap(rawSet))) { return true; }
-
-		return false;
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/reset/HBaseResetMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/reset/HBaseResetMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/reset/HBaseResetMapper.java	(nonexistent)
@@ -1,127 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.reset;
-
-import java.io.IOException;
-
-import org.apache.hadoop.hbase.KeyValue;
-import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Mutation;
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-
-import com.google.common.collect.Lists;
-import com.google.protobuf.GeneratedMessage;
-import com.google.protobuf.InvalidProtocolBufferException;
-
-import eu.dnetlib.data.mapreduce.hbase.HBaseTableUtils.VolatileColumnFamily;
-import eu.dnetlib.data.mapreduce.util.DedupUtils;
-import eu.dnetlib.data.proto.FieldTypeProtos.DataInfo;
-import eu.dnetlib.data.proto.OafProtos.Oaf;
-import eu.dnetlib.data.proto.RelTypeProtos.SubRelType;
-
-public class HBaseResetMapper extends TableMapper<ImmutableBytesWritable, Mutation> {
-
-	private static final boolean WRITE_TO_WAL = false;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		super.setup(context);
-
-		String table = null;
-		for (String s : Lists.newArrayList("hbase.mapreduce.inputtable", "hbase.mapred.inputtable")) {
-			try {
-				table = context.getConfiguration().get(s).trim();
-			} catch (NullPointerException e) {}
-		}
-		if (table == null) throw new IllegalStateException("unable to find table name");
-		System.out.println("I start to reset table '" + table + "'");
-	}
-
-	@Override
-	protected void map(final ImmutableBytesWritable key, final Result value, final Context context) throws IOException, InterruptedException {
-		// System.out.println("EVALUATING " + Bytes.toString(key.get()));
-		byte[] bKey = key.copyBytes();
-
-		if (DedupUtils.isRoot(new String(bKey))) {
-			deleteRow(key, context);
-			return;
-		}
-
-		for (KeyValue kv : value.list()) {
-			if (Bytes.toString(kv.getQualifier()).equals(DedupUtils.BODY_S) || isColumn(kv.getFamily())) {
-				Oaf oaf = Oaf.parseFrom(kv.getValue());
-				if (isColumnToUpdate(oaf)) {
-					updateColumn(key, kv, resetOaf(oaf), context);
-				}
-			} else if (isColumnToDelete(kv)) {
-				deleteColumn(key, kv, context);
-			}
-		}
-	}
-
-	private boolean isColumn(final byte[] family) {
-		try {
-			String s = Bytes.toString(family);
-			return !s.contains(SubRelType.dedup.toString()) && !s.contains(SubRelType.dedupSimilarity.toString()) && !VolatileColumnFamily.isVolatile(s)
-					&& (DedupUtils.decodeCF(family) != null);
-		} catch (IllegalArgumentException e) {
-			return false;
-		}
-	}
-
-	private boolean isColumnToUpdate(final Oaf oaf) {
-		return oaf.getDataInfo().hasDeletedbyinference() && oaf.getDataInfo().getDeletedbyinference();
-	}
-
-	private Oaf resetOaf(final Oaf oaf) {
-		// I reset deletedbyinference
-		DataInfo.Builder dataInfoBuilder = DataInfo.newBuilder(oaf.getDataInfo());
-		dataInfoBuilder.setDeletedbyinference(false).setInferenceprovenance("");
-		Oaf.Builder oafBuilder = Oaf.newBuilder(oaf);
-		oafBuilder.setDataInfo(dataInfoBuilder);
-		return oafBuilder.build();
-	}
-
-	private boolean isColumnToDelete(final KeyValue kv) throws InvalidProtocolBufferException {
-		// Columns in "dedupRel" and "similarRel" column families and with the attribute inferred=true must be deleted
-		final String cf = Bytes.toString(kv.getFamily());
-
-		if (cf.contains(SubRelType.dedup.toString()) || cf.contains(SubRelType.dedupSimilarity.toString()) || VolatileColumnFamily.isVolatile(cf)) return true;
-
-		final Oaf oaf = Oaf.parseFrom(kv.getValue());
-		return oaf.getDataInfo().hasInferred() && oaf.getDataInfo().getInferred();
-	}
-
-	private void updateColumn(final ImmutableBytesWritable key, final KeyValue col, final GeneratedMessage newValue, final Context context) throws IOException,
-			InterruptedException {
-		byte[] bKey = key.copyBytes();
-		Put put = new Put(bKey);
-		put.add(col.getFamily(), col.getQualifier(), newValue.toByteArray());
-		put.setWriteToWAL(WRITE_TO_WAL);
-		context.write(key, put);
-		context.getCounter("reset job", "update body").increment(1);
-		// System.out.println("   --- UPDATED ROW key=" + Bytes.toString(bKey));
-	}
-
-	private void deleteRow(final ImmutableBytesWritable key, final Context context) throws IOException, InterruptedException {
-		byte[] bKey = key.copyBytes();
-		Delete d = new Delete(bKey);
-		d.setWriteToWAL(WRITE_TO_WAL);
-		context.write(key, d);
-		context.getCounter("reset job", "delete row").increment(1);
-		// System.out.println("   --- DELETED ROW key=" + Bytes.toString(bKey));
-	}
-
-	private void deleteColumn(final ImmutableBytesWritable key, final KeyValue col, final Context context) throws IOException, InterruptedException {
-		byte[] bKey = key.copyBytes();
-		Delete d = new Delete(bKey);
-		d.deleteColumns(col.getFamily(), col.getQualifier());
-		d.setWriteToWAL(WRITE_TO_WAL);
-		context.write(key, d);
-		context.getCounter("reset job", "delete column").increment(1);
-		// System.out.println("   --- DELETED COLUMN key=" + Bytes.toString(bKey) + " cf=" + Bytes.toString(col.getFamily()) + ", q="
-		// + Bytes.toString(col.getQualifier()));
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/ImportRecordsMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/ImportRecordsMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/ImportRecordsMapper.java	(nonexistent)
@@ -1,74 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dataimport;
-
-import java.io.IOException;
-
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Mapper;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.transform.Column;
-import eu.dnetlib.data.transform.Row;
-import eu.dnetlib.data.transform.XsltRowTransformer;
-import eu.dnetlib.data.transform.XsltRowTransformerFactory;
-
-public class ImportRecordsMapper extends Mapper<Text, Text, ImmutableBytesWritable, Put> {
-
-	/**
-	 * logger.
-	 */
-	private static final Log log = LogFactory.getLog(ImportRecordsMapper.class); // NOPMD by marko on 11/24/08 5:02 PM
-
-	private static final boolean WRITE_TO_WAL = false;
-
-	private XsltRowTransformer transformer;
-
-	private String xslt;
-
-	private ImmutableBytesWritable ibw;
-
-	@Override
-	protected void setup(final Context context) throws IOException, InterruptedException {
-		super.setup(context);
-
-		this.xslt = context.getConfiguration().get(JobParams.HBASE_IMPORT_XSLT).trim();
-
-		if ((xslt == null) || xslt.isEmpty()) throw new IllegalArgumentException("missing xslt");
-
-		transformer = new XsltRowTransformerFactory().getTransformer(xslt);
-		ibw = new ImmutableBytesWritable();
-
-		log.info("got xslt: '" + xslt);
-		log.info("using trasformer: '" + transformer.getTransformerClassName() + "'");
-	}
-
-	@Override
-	protected void map(final Text key, final Text value, final Context context) throws IOException, InterruptedException {
-		try {
-			for (final Row row : transformer.apply(value.toString())) {
-
-				final byte[] rowKey = Bytes.toBytes(row.getKey());
-				final Put put = new Put(rowKey);
-				put.setWriteToWAL(WRITE_TO_WAL);
-
-				for (final Column<String, byte[]> col : row) {
-					final byte[] family = Bytes.toBytes(row.getColumnFamily());
-					final byte[] qualifier = Bytes.toBytes(col.getName());
-					put.add(family, qualifier, col.getValue());
-				}
-				ibw.set(rowKey);
-				context.write(ibw, put);
-				context.getCounter("mdstore", row.getColumnFamily()).increment(row.getColumns().size());
-			}
-		} catch (final Throwable e) {
-			log.error("error importing the following record on HBase: " + value.toString(), e);
-			context.getCounter("error", e.getClass().getName()).increment(1);
-			throw new RuntimeException(e);
-		}
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/HTableImportJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/HTableImportJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/HTableImportJob.java	(nonexistent)
@@ -1,32 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dataimport;
-
-import java.util.Properties;
-
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableOutputFormat;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.hbase.AbstractHBaseMapReduceJob;
-
-public class HTableImportJob extends AbstractHBaseMapReduceJob {
-
-	@Override
-	protected Job setJobDetails(Job job, Properties p) throws Exception {
-		job.setMapOutputKeyClass(ImmutableBytesWritable.class);
-		job.setMapOutputValueClass(Put.class);
-		job.setMapperClass(SequenceFileRecordMapper.class);
-
-		job.setInputFormatClass(SequenceFileInputFormat.class);
-		job.setOutputFormatClass(TableOutputFormat.class);
-
-		SequenceFileInputFormat.setInputPaths(job, p.getProperty(JobParams.MAPRED_INPUT_DIR));
-
-		job.setNumReduceTasks(0);
-
-		return job;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/ImportRecordsJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/ImportRecordsJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/ImportRecordsJob.java	(nonexistent)
@@ -1,35 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dataimport;
-
-import java.util.Properties;
-
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableOutputFormat;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.hbase.AbstractHBaseMapReduceJob;
-
-public class ImportRecordsJob extends AbstractHBaseMapReduceJob {
-
-	@Override
-	protected Job setJobDetails(Job job, Properties p) throws Exception {
-		job.setMapOutputKeyClass(ImmutableBytesWritable.class);
-		job.setMapOutputValueClass(Put.class);
-		job.setMapperClass(ImportRecordsMapper.class);
-
-		job.setInputFormatClass(SequenceFileInputFormat.class);
-		job.setOutputFormatClass(TableOutputFormat.class);
-
-		SequenceFileInputFormat.setInputPaths(job, p.getProperty(JobParams.MAPRED_INPUT_DIR));
-
-		job.getConfiguration().setBoolean("mapred.map.tasks.speculative.execution", false);
-		job.getConfiguration().setBoolean("mapreduce.map.speculative", false);
-
-		job.setNumReduceTasks(0);
-
-		return job;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/DeleteRecordsMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/DeleteRecordsMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/DeleteRecordsMapper.java	(nonexistent)
@@ -1,72 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dataimport;
-
-import java.io.IOException;
-import java.util.Set;
-
-import org.apache.hadoop.hbase.KeyValue;
-import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapper;
-import org.apache.hadoop.hbase.util.Bytes;
-
-import com.google.common.base.Splitter;
-import com.google.common.collect.Sets;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-
-public class DeleteRecordsMapper extends TableMapper<ImmutableBytesWritable, Delete> {
-
-	private Set<String> delPrefix;
-
-	@Override
-	protected void setup(Context context) throws IOException, InterruptedException {
-		super.setup(context);
-		delPrefix = extractPrefixes(context.getConfiguration().get("nsPrefix"));
-
-		String table = context.getConfiguration().get(JobParams.HBASE_SOURCE_TABLE).trim();
-
-		System.out.println("I start to delete records (table=" + table + ", prefixes=" + delPrefix);
-	}
-
-	@Override
-	protected void map(ImmutableBytesWritable key, Result value, Context context) throws IOException, InterruptedException {
-		byte[] bkey = key.get();
-
-		System.out.println("EVALUATING " + Bytes.toString(bkey));
-
-		if (testValue(bkey)) { // If key contains repoId I delete the row
-			context.write(key, new Delete(bkey));
-			System.out.println("   --- DELETED");
-			context.getCounter("MDStore Cleaner", "Deleted records").increment(1);
-		} else { // otherwise I try to delete the columns containing repoId 
-			for (KeyValue kv : value.list()) {
-				byte[] col = kv.getQualifier();
-				if (testValue(col)) {
-					Delete d = new Delete(bkey);
-					d.deleteColumns(kv.getFamily(), col);
-					context.write(key, d);
-					System.out.println("   --- DELETED COLUMN: " + Bytes.toString(col));
-					context.getCounter("MDStore Cleaner", "Deleted columns").increment(1);
-				}
-			}
-		}
-	}
-
-	private Set<String> extractPrefixes(String s) {
-		return Sets.newHashSet(Splitter.on(",").trimResults().omitEmptyStrings().split(s));
-	}
-
-	private boolean testValue(byte[] bkey) {
-		String s = Bytes.toString(bkey);
-
-		if (s.contains("|") && s.contains("::")) {
-			return delPrefix.contains(s.substring(s.indexOf("|") + 1, s.indexOf("::")));
-		} else if (s.contains("::")) {
-			return delPrefix.contains(s.substring(0, s.indexOf("::")));
-		} else {
-			return false;
-		}
-
-	}
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/SequenceFileRecordMapper.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/SequenceFileRecordMapper.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/SequenceFileRecordMapper.java	(nonexistent)
@@ -1,38 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dataimport;
-
-import java.io.IOException;
-
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Mapper;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-
-public class SequenceFileRecordMapper extends Mapper<Text, Text, ImmutableBytesWritable, Put> {
-
-	private String family;
-
-	private String qualifier;
-	
-	@Override
-	protected void setup(Context context) throws IOException, InterruptedException {
-		super.setup(context);
-		
-		this.family = context.getConfiguration().get(JobParams.HBASE_TARGET_FAMILY).trim();
-		this.qualifier = context.getConfiguration().get(JobParams.HBASE_TARGET_QUALIFIER).trim();
-	}
-	
-	@Override
-	protected void map(Text key, Text value, Context context) throws IOException, InterruptedException {
-
-		//byte[] rowKey = DigestUtils.md5(key.toString());
-		byte[] rowKey = Bytes.toBytes(key.toString());
-		
-		Put put = new Put(rowKey).add(Bytes.toBytes(family), Bytes.toBytes(qualifier), Bytes.toBytes(value.toString()));
-	
-        context.write(new ImmutableBytesWritable(rowKey), put);
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/HFileImportJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/HFileImportJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/HFileImportJob.java	(nonexistent)
@@ -1,46 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dataimport;
-
-import java.util.Properties;
-
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.client.HTable;
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.HFileOutputFormat;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
-import org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.hbase.AbstractHBaseMapReduceJob;
-
-public class HFileImportJob extends AbstractHBaseMapReduceJob {
-
-	@Override
-	protected Job setJobDetails(Job job, Properties p) throws Exception {
-		job.setMapOutputKeyClass(ImmutableBytesWritable.class);
-		job.setMapOutputValueClass(Put.class);
-		//job.setMapOutputValueClass(KeyValue.class);
-		job.setMapperClass(ImportRecordsMapper.class);
-
-		job.setInputFormatClass(SequenceFileInputFormat.class);
-		job.setOutputFormatClass(HFileOutputFormat.class);
-
-		job.setPartitionerClass(TotalOrderPartitioner.class);
-
-		SequenceFileInputFormat.setInputPaths(job, p.getProperty(JobParams.MAPRED_INPUT_DIR));
-
-		HFileOutputFormat.setOutputPath(job, new Path(p.getProperty(JobParams.MAPRED_OUTPUT_DIR)));
-		HFileOutputFormat.setCompressOutput(job, true);
-
-		final Configuration conf = HBaseConfiguration.create(job.getConfiguration());
-		final String tableName = p.getProperty(JobParams.HBASE_TARGET_TABLE);
-
-		HFileOutputFormat.configureIncrementalLoad(job, new HTable(conf, tableName));
-
-		return job;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/DeleteRecordsJob.java
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/DeleteRecordsJob.java	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/src/main/java/eu/dnetlib/data/mapreduce/hbase/dataimport/DeleteRecordsJob.java	(nonexistent)
@@ -1,51 +0,0 @@
-package eu.dnetlib.data.mapreduce.hbase.dataimport;
-
-import java.io.IOException;
-import java.util.Properties;
-
-import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Scan;
-import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
-import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
-import org.apache.hadoop.hbase.mapreduce.TableOutputFormat;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.mapreduce.Job;
-
-import eu.dnetlib.data.mapreduce.JobParams;
-import eu.dnetlib.data.mapreduce.hbase.AbstractHBaseMapReduceJob;
-
-public class DeleteRecordsJob extends AbstractHBaseMapReduceJob {
-
-	@Override
-	protected Job setJobDetails(Job job, Properties p) throws Exception {
-
-		initMapper(job, getScan(p), p.getProperty(JobParams.HBASE_SOURCE_TABLE));
-
-		job.setOutputFormatClass(TableOutputFormat.class);
-		job.setMapOutputKeyClass(Text.class);
-		job.setMapOutputValueClass(Delete.class);
-		job.setNumReduceTasks(0);
-
-		job.getConfiguration().setBoolean("mapred.map.tasks.speculative.execution", false);
-		job.getConfiguration().setBoolean("mapreduce.map.speculative", false);
-
-		return job;
-	}
-
-	private void initMapper(final Job job, final Scan scan, final String sourceTable) {
-		try {
-			TableMapReduceUtil.initTableMapperJob(sourceTable, scan, DeleteRecordsMapper.class, Text.class, ImmutableBytesWritable.class, job);
-		} catch (IOException e) {
-			throw new RuntimeException(e);
-		}
-	}
-
-	private Scan getScan(Properties p) {
-		Scan scan = new Scan();
-		scan.setCaching(500); // 1 is the default in Scan, which will be bad for MapReduce jobs
-		scan.setCacheBlocks(false); // don't set to true for MR jobs
-
-		return scan;
-	}
-
-}
Index: modules/dnet-mapreduce-jobs/branches/yarn/pom.xml
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/pom.xml	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/pom.xml	(nonexistent)
@@ -1,235 +0,0 @@
-<?xml version="1.0" ?>
-<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
-	<parent>
-		<groupId>eu.dnetlib</groupId>
-		<artifactId>dnet-hadoop-parent</artifactId>
-		<version>1.0.1-CDH-5.3.0-SNAPSHOT</version>
-		<relativePath />
-	</parent>
-	<modelVersion>4.0.0</modelVersion>
-	<groupId>eu.dnetlib</groupId>
-	<artifactId>dnet-mapreduce-jobs</artifactId>
-	<version>0.0.6.6-CDH-5.3.0-SNAPSHOT</version>
-	<packaging>jar</packaging>
-	<scm>
-		<developerConnection>scm:svn:https://svn.driver.research-infrastructures.eu/driver/dnet40/modules/dnet-mapreduce-jobs/trunk</developerConnection>
-	</scm>
-	<build>
-		<plugins>
-			<plugin>
-				<artifactId>maven-assembly-plugin</artifactId>
-				<configuration>
-					<archive>
-						<manifest>
-							<mainClass>eu.dnetlib.data.mapreduce.hbase.dataimport.ImportRecordsJob</mainClass>
-						</manifest>
-					</archive>
-					<descriptorRefs>
-						<descriptorRef>jar-with-dependencies</descriptorRef>
-					</descriptorRefs>
-				</configuration>
-			</plugin>
-		</plugins>
-	</build>
-	<repositories>
-		<!-- Cloudera Repositories -->
-		<repository>
-			<snapshots>
-				<enabled>false</enabled>
-			</snapshots>
-			<id>cloudera-central</id>
-			<name>cloundera-libs-release</name>
-			<url>http://maven.research-infrastructures.eu/nexus/content/repositories/cloudera-central</url>
-		</repository>
-		<repository>
-			<id>cloudera-snapshots</id>
-			<name>cloudera-libs-snapshot</name>
-			<url>http://maven.research-infrastructures.eu/nexus/content/repositories/cloudera-snapshots</url>
-		</repository>
-		<repository>
-			<id>typesafe</id>
-			<name>typesafe-releases</name>
-			<url>http://maven.research-infrastructures.eu/nexus/content/repositories/typesafe</url>
-		</repository>
-	</repositories>
-	<dependencies>
-		<dependency>
-			<groupId>commons-logging</groupId>
-			<artifactId>commons-logging</artifactId>
-			<version>${commons.logging.version}</version>
-		</dependency>
-		<dependency>
-			<groupId>junit</groupId>
-			<artifactId>junit</artifactId>
-			<version>${junit.version}</version>
-			<scope>test</scope>
-		</dependency>
-		<dependency>
-			<groupId>com.google.code.gson</groupId>
-			<artifactId>gson</artifactId>
-			<version>${google.gson.version}</version>
-		</dependency>
-		<dependency>
-			<groupId>eu.dnetlib</groupId>
-			<artifactId>dnet-index-solr-common</artifactId>
-			<version>[1.0.0,2.0.0)</version>
-		</dependency>
-		<dependency>
-			<groupId>org.apache.solr</groupId>
-			<artifactId>solr-solrj</artifactId>
-			<version>${apache.solr.version}</version>
-			<exclusions>
-				<exclusion>
-					<artifactId>wstx-asl</artifactId>
-					<groupId>org.codehaus.woodstox</groupId>
-				</exclusion>
-				<exclusion>
-					<artifactId>httpcore</artifactId>
-					<groupId>org.apache.httpcomponents</groupId>
-				</exclusion>
-				<exclusion>
-					<artifactId>httpclient</artifactId>
-					<groupId>org.apache.httpcomponents</groupId>
-				</exclusion>
-			</exclusions>
-		</dependency>
-		<dependency>
-			<groupId>com.mycila</groupId>
-			<artifactId>xmltool</artifactId>
-			<version>3.3</version>
-		</dependency>
-		<dependency>
-			<groupId>eu.dnetlib</groupId>
-			<artifactId>cnr-misc-utils</artifactId>
-			<version>[1.0.0,2.0.0)</version>
-			<exclusions>
-				<exclusion>
-					<groupId>apache</groupId>
-					<artifactId>commons-lang</artifactId>
-				</exclusion>
-			</exclusions>
-		</dependency>
-		<dependency>
-			<groupId>eu.dnetlib</groupId>
-			<artifactId>dnet-mapreduce-submitter</artifactId>
-			<version>[2.0.0,3.0.0)</version>
-			<exclusions>
-				<exclusion>
-					<artifactId>commons-httpclient</artifactId>
-					<groupId>commons-httpclient</groupId>
-				</exclusion>
-			</exclusions>
-		</dependency>
-		<dependency>
-			<groupId>org.apache.hbase</groupId>
-			<artifactId>hbase-server</artifactId>
-			<version>${apache.hbase.version}</version>
-			<exclusions>
-				<exclusion>
-					<artifactId>hadoop-core</artifactId>
-					<groupId>org.apache.hadoop</groupId>
-				</exclusion>
-				<exclusion>
-					<artifactId>org.mortbay.jetty</artifactId>
-					<groupId>servlet-api</groupId>
-				</exclusion>
-				<exclusion>
-					<artifactId>javax.servlet.jsp</artifactId>
-					<groupId>jsp-api</groupId>
-				</exclusion>
-				<exclusion>
-					<artifactId>javax.servlet</artifactId>
-					<groupId>servlet-api</groupId>
-				</exclusion>
-				<exclusion>
-					<artifactId>jasper-compiler</artifactId>
-					<groupId>tomcat</groupId>
-				</exclusion>
-				<exclusion>
-					<artifactId>jasper-runtime</artifactId>
-					<groupId>tomcat</groupId>
-				</exclusion>
-				<exclusion>
-					<artifactId>jsp-api-2.1</artifactId>
-					<groupId>org.mortbay.jetty</groupId>
-				</exclusion>
-				<exclusion>
-					<artifactId>jsp-api</artifactId>
-					<groupId>javax.servlet.jsp</groupId>
-				</exclusion>
-				<exclusion>
-					<artifactId>servlet-api</artifactId>
-					<groupId>javax.servlet</groupId>
-				</exclusion>
-				<exclusion>
-					<artifactId>servlet-api-2.5</artifactId>
-					<groupId>org.mortbay.jetty</groupId>
-				</exclusion>
-			</exclusions>
-		</dependency>
-		<dependency>
-			<groupId>eu.dnetlib</groupId>
-			<artifactId>dnet-openaire-data-protos</artifactId>
-			<version>[3.0.0,4.0.0)</version>
-		</dependency>
-		<dependency>
-			<groupId>com.googlecode.protobuf-java-format</groupId>
-			<artifactId>protobuf-java-format</artifactId>
-			<version>1.2</version>
-		</dependency>
-		<dependency>
-			<groupId>eu.dnetlib</groupId>
-			<artifactId>dnet-openaireplus-mapping-utils</artifactId>
-			<version>[3.0.0,4.0.0)</version>
-		</dependency>
-		<dependency>
-			<groupId>org.antlr</groupId>
-			<artifactId>stringtemplate</artifactId>
-			<version>3.2</version>
-		</dependency>
-		<dependency>
-			<groupId>org.json</groupId>
-			<artifactId>json</artifactId>
-			<version>20140107</version>
-		</dependency>
-		<dependency>
-			<groupId>com.typesafe</groupId>
-			<artifactId>config</artifactId>
-			<version>1.2.1</version>
-		</dependency>
-		<dependency>
-			<groupId>eu.dnetlib</groupId>
-			<artifactId>dnet-pace-core</artifactId>
-			<version>[2.0.0,3.0.0)</version>
-		</dependency>
-		<dependency>
-			<groupId>org.apache.httpcomponents</groupId>
-			<artifactId>httpclient</artifactId>
-			<version>4.2.3</version>
-		</dependency>
-		<dependency>
-			<groupId>org.mongodb</groupId>
-			<artifactId>mongo-java-driver</artifactId>
-			<version>${mongodb.driver.version}</version>
-		</dependency>
-		<dependency>
-			<groupId>eu.dnetlib</groupId>
-			<artifactId>dnet-actionmanager-common</artifactId>
-			<version>[2.0.0,3.0.0)</version>
-		</dependency>
-
-
-		<dependency>
-			<groupId>org.elasticsearch</groupId>
-			<artifactId>elasticsearch-hadoop-mr</artifactId>
-			<version>2.0.2</version>
-		</dependency>
-		<dependency>
-			<groupId>org.mockito</groupId>
-			<artifactId>mockito-all</artifactId>
-			<version>1.8.5</version>
-		</dependency>
-
-	</dependencies>
-</project>
Index: modules/dnet-mapreduce-jobs/branches/yarn/update.sh
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/update.sh	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/update.sh	(nonexistent)
@@ -1,4 +0,0 @@
-#!/bin/sh
-
-cd target/classes/
-zip ~/.m2/repository/eu/dnetlib/dnet-mapreduce-jobs-assembly/0.0.6.5-SNAPSHOT/dnet-mapreduce-jobs-assembly-0.0.6.5-SNAPSHOT.jar $(find eu -name '*.class')

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/update.sh
___________________________________________________________________
Deleted: svn:executable
## -1 +0,0 ##
-*
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/upload.sh
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/upload.sh	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/upload.sh	(nonexistent)
@@ -1,13 +0,0 @@
-#!/bin/bash
-
-echo "copy uber jar to hadoop node..."
-scp ~/.m2/repository/eu/dnetlib/dnet-mapreduce-jobs-assembly/0.0.6.6-SNAPSHOT/dnet-mapreduce-jobs-assembly-0.0.6.6-SNAPSHOT.jar root@quorum2.t.hadoop.research-infrastructures.eu:/tmp/
-
-echo "copy done! Now uploading library..."
-
-ssh root@quorum2.t.hadoop.research-infrastructures.eu su hdfs <<'EOF'
-export HADOOP_LOG_DIR=/var/log/hadoop-0.20-mapreduce
-hadoop fs -rm -skipTrash hdfs://nmis-hadoop-cluster/user/dnet/lib/dnet-mapreduce-jobs-assembly-0.0.6.6-SNAPSHOT.jar
-hadoop fs -copyFromLocal /tmp/dnet-mapreduce-jobs-assembly-0.0.6.6-SNAPSHOT.jar hdfs://nmis-hadoop-cluster/user/dnet/lib/
-echo "Done!"
-EOF
\ No newline at end of file

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/upload.sh
___________________________________________________________________
Deleted: svn:executable
## -1 +0,0 ##
-*
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn/install.sh
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn/install.sh	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn/install.sh	(nonexistent)
@@ -1,5 +0,0 @@
-#!/bin/bash
-
-mvn clean install -DskipTests=true;
-rm -rf ~/.m2/repository/eu/dnetlib/dnet-mapreduce-jobs-assembly;
-mvn assembly:assembly -DskipTests=true && mvn install:install-file -Dfile=target/dnet-mapreduce-jobs-0.0.6.6-SNAPSHOT-jar-with-dependencies.jar -DgroupId=eu.dnetlib -DartifactId=dnet-mapreduce-jobs-assembly -Dversion=0.0.6.6-SNAPSHOT -Dpackaging=jar
\ No newline at end of file

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn/install.sh
___________________________________________________________________
Deleted: svn:executable
## -1 +0,0 ##
-*
\ No newline at end of property
Index: modules/dnet-mapreduce-jobs/branches/yarn
===================================================================
--- modules/dnet-mapreduce-jobs/branches/yarn	(revision 51209)
+++ modules/dnet-mapreduce-jobs/branches/yarn	(nonexistent)

Property changes on: modules/dnet-mapreduce-jobs/branches/yarn
___________________________________________________________________
Deleted: svn:ignore
## -1,7 +0,0 ##
-.classpath
-.settings
-target
-.project
-
-
-*.iml
