D-Net

<?xml version="1.0" encoding="UTF-8"?>

<de:comments xmlns:de="http://de.tukl.softech.agileReview">

  <de:author name="mafju"/>

  <de:files>

    <de:project name="icm-iis-export-hbase">

      <de:folder name="src">

        <de:folder name="main">

          <de:folder name="resources">

            <de:folder name="eu">

              <de:folder name="dnetlib">

                <de:folder name="iis">

                  <de:folder name="export">

                    <de:folder name="hbase">

                      <de:folder name="oozie_app">

                        <de:file name="workflow.xml">

                          <de:comment id="c0" author="mafju" reviewID="2013-03-08 export data generator" creation-date="2013-03-08T11:22:35.759+01:00" last-modified="2013-03-11T15:06:14.902+01:00" priority="0" recipient="mhorst" status="1" revision="0">

                            <de:text>I would personally use more descriptive names for data stores e.g. "document_with_inferenced_data" instead of less readable "doc_with_inf_data". Especially "inf" word is ambiguous because it could also mean, e.g. "infinite", "inferior".</de:text>

                            <de:replies>

                              <de:reply author="mhorst" creation-date="2013-03-11T14:21:52.972+01:00">I will rename it with full names.</de:reply>

                              <de:reply author="mhorst" creation-date="2013-03-11T15:03:58.209+01:00">Renamed, closing.</de:reply>

                            </de:replies>

                          </de:comment>

                        </de:file>

                      </de:folder>

                      <de:file name="job.properties">

                        <de:comment id="c4" author="mafju" reviewID="2013-03-08 export data generator" creation-date="2013-03-08T12:59:58.747+01:00" last-modified="2013-03-11T15:07:27.033+01:00" priority="0" recipient="mhorst" status="1" revision="0">

                          <de:text>It seems that there's a convention that the data that is not available is marked with "x". This does not seem to be elegant since it is easy to confuse the special marker "x" with a real directory.</de:text>

                          <de:replies>

                            <de:reply author="mhorst" creation-date="2013-03-08T20:25:22.582+01:00">Come on, this was clearly not intended to be considered as a final solution. Just a quick and dirty 'blank' to make things working asap. Obviously creating 'x' directory in current folder and placing JSON files will cause errors, but I was confident I would not do this ;)

I am not sure what would be the best 'blank' value anyway,  empty string seem to cause errors.</de:reply>

                            <de:reply author="mafju" creation-date="2013-03-11T13:25:17.949+01:00">> Come on, this was clearly not intended to be considered as a final solution. Just a quick and dirty 'blank' to make things working asap.

:D How would I know that? I thought this was also something I should review; business as usual :) Maybe some "FIXME" comment or something similar would be in order?

> I am not sure what would be the best 'blank' value anyway,  empty string seem to cause errors.

Maybe something that could not possibly be a valid path in file system, e.g. "*"?</de:reply>

                            <de:reply author="mhorst" creation-date="2013-03-11T14:19:30.057+01:00">I am not sure how would '*' as directory path be handled, what I would like to achieve is to have assurance of providing empty directory without any files.

I've got simple idea: at prepare state create export_data_gen/empty directory and provide this path as default input :)</de:reply>

                            <de:reply author="mhorst" creation-date="2013-03-11T14:51:12.980+01:00">Confirmed: creating empty dir and setting values to:

${workingDir}/export_data_gen/empty/

works as expected.</de:reply>

                            <de:reply author="mafju" creation-date="2013-03-11T15:07:27.033+01:00">Nice</de:reply>

                          </de:replies>

                        </de:comment>

                        <de:comment id="c5" author="mafju" reviewID="2013-03-08 export data generator" creation-date="2013-03-08T13:04:17.272+01:00" last-modified="2013-03-11T13:26:05.253+01:00" priority="0" recipient="mhorst" status="1" revision="0">

                          <de:text>There's a hardcoded path. Is this done intentionally?</de:text>

                          <de:replies>

                            <de:reply author="mhorst" creation-date="2013-03-08T20:29:15.900+01:00">Obviously, yes. As I had written before on some other comment: this module, and all of its configuration was intended to be used by me only. So it does not follow conventions and can be ugly.

If it will be prepared to be used by others, all hardcoded values will be replaced by placeholders or proper default values.</de:reply>

                            <de:reply author="mafju" creation-date="2013-03-11T13:25:50.147+01:00">Ok</de:reply>

                          </de:replies>

                        </de:comment>

                      </de:file>

                    </de:folder>

                  </de:folder>

                </de:folder>

              </de:folder>

            </de:folder>

          </de:folder>

          <de:folder name="java">

            <de:folder name="eu">

              <de:folder name="dnetlib">

                <de:folder name="iis">

                  <de:folder name="export">

                    <de:folder name="hbase">

                      <de:folder name="generator">

                        <de:file name="JsonBasedInferencedDataGenerator.java">

                          <de:comment id="c1" author="mafju" reviewID="2013-03-08 export data generator" creation-date="2013-03-08T12:14:23.711+01:00" last-modified="2013-03-11T13:18:37.576+01:00" priority="0" recipient="mhorst" status="1" revision="0">

                            <de:text>The logic of code in this class is quite complicated, its not clear what it does when just glancing at it. If it is going to be used by some other people, maybe some javadoc would be appropriate, or even better: simplifying the logic (if possible)?</de:text>

                            <de:replies>

                              <de:reply author="mhorst" creation-date="2013-03-08T19:31:34.656+01:00">It wasn't intended for other people, I am the one struggling with export so I was mainly thinking about myself when developing this module.</de:reply>

                              <de:reply author="mafju" creation-date="2013-03-11T13:07:23.015+01:00">Fair enough, although maybe in the future someone would want to use it.</de:reply>

                            </de:replies>

                          </de:comment>

                          <de:comment id="c2" author="mafju" reviewID="2013-03-08 export data generator" creation-date="2013-03-08T12:27:46.775+01:00" last-modified="2013-03-11T14:14:05.659+01:00" priority="0" recipient="mhorst" status="1" revision="0">

                            <de:text>This is not clear to me. The name of the "modeToPathMap" object suggests that we're extracting from the "parameters" an element whose kay/name is equal to some path. This is probably not something that we're really doing?

BTW, we're passing here some paths do files through parameters of the worklfow node, but a standard way to do it is to pass such information through input ports. Why do we use parameters? Maybe it would be more appropriate (and more explicit) to use the input ports?</de:text>

                            <de:replies>

                              <de:reply author="mhorst" creation-date="2013-03-08T20:07:43.964+01:00">> This is not clear to me. The name of the "modeToPathMap" object suggests that we're extracting from the "parameters" an element whose kay/name is equal to some path. This is probably not something that we're really doing?

This is required to associate export mode with proper input path and to make it easier to handle individual modes using single handleMode() method in the loop.

> BTW, we're passing here some paths do files through parameters of the worklfow node, but a standard way to do it is to pass such information through input ports. Why do we use parameters? Maybe it would be more appropriate (and more explicit) to use the input ports?

The main reason for using parameters over ports was their less restrictive nature in terms of requirement for providing given path explicitly as in most cases only single datastore will be generated. When using ports: all of them have to be provided in configuration. I will switch back to ports because I had to provide all placeholders in job.properties anyway so I gained nothing due to configuration restrictions.</de:reply>

                              <de:reply author="mafju" creation-date="2013-03-11T13:17:24.538+01:00">> The main reason for using parameters over ports was their less restrictive nature in terms of requirement for providing given path explicitly as in most cases only single datastore will be generated. When using ports: all of them have to be provided in configuration. I will switch back to ports because I had to provide all placeholders in job.properties anyway so I gained nothing due to configuration restrictions.

Ok. If it wouldn't make much difference, you could also generate all possible datastores every time this workflow node is run and then use only some of them in the next workflow node.</de:reply>

                              <de:reply author="mhorst" creation-date="2013-03-11T14:14:05.659+01:00">True, but the main problem was not the requirement for generating output for all 4 ports but the neccessity of providing all 4 input ports with JSONs. In most cases I would like to provide JSONs for single port only (e.g. similarities when testing similarity export).</de:reply>

                            </de:replies>

                          </de:comment>

                          <de:comment id="c3" author="mafju" reviewID="2013-03-08 export data generator" creation-date="2013-03-08T12:36:39.418+01:00" last-modified="2013-03-11T13:18:30.472+01:00" priority="0" recipient="mhorst" status="2" revision="0">

                            <de:text>Is doing this check necessary? Don't we assume here that the "modeToPathMap" contains all possible "ExportModes"? If so, double checking it here does not seem to be necessary.</de:text>

                            <de:replies>

                              <de:reply author="mhorst" creation-date="2013-03-08T20:14:06.373+01:00">This might seem to be excessive although when someone changes list of iterable modes without filling in the modeToPathMap this will trigger meaningful error message. 

Now I realized we could simply iterate over modeToPathMap keys and skip this condition :)</de:reply>

                            </de:replies>

                          </de:comment>

                        </de:file>

                      </de:folder>

                    </de:folder>

                  </de:folder>

                </de:folder>

              </de:folder>

            </de:folder>

          </de:folder>

        </de:folder>

      </de:folder>

    </de:project>

  </de:files>

</de:comments>