openmainframeproject / ade Goto Github PK

This issue is raised in reference to JOSS submission 3052.

The writing style in the paper is colloquial; in particular, the sections Statement of Need and Novelties. The language is understandable (I enjoyed reading the paper!), but it's suited more for a magazine article or a blog post. Since JOSS is an academic journal, the language needs to be more precise. The paper in general, and the sections mentioned above in particular, should be edited to reduce verbosity and use more precise language.

Hi,

First set of comments for openjournals/joss-reviews#2972

Regarding the paper:

• can this submission be considered a "research software"?
  I see 3 cited papers in the references, but none of the authors?
  Has this software been used in any published research work?

• "Our approach is very different: we treat this as a statistical problem and avoid introducing computationally heavy deep learning into the picture. This makes the solution more explainable and faster to compute".
  Is there any published empirical evidence to support this claim?

• out of the 3 references, only 1 is peer-reviewed.
  The other 2 are from arXiv.
  Can you please add some more peer-reviewed references of related work?
  In particular, add references to existing open-source software that is doing something similar to your work, or if there is none, explicitly state it.

• "Novelties: ADE is written in the Java language that makes it ideal to be used across any platform without changing the underlying code". Why does it matter? You are targeting Linux. Furthermore, a lot of other languages are cross-platform (C#, JavaScript, Python, etc.)

• "Novelties: ADE comes batteries-included: One can just build the binary...". Sorry, but "batteries-included" and "just build the binary" do not go well together...

• "Novelties: Minimal external dependencies". Why does it matter? You are releasing a self-contained tool, not a library.

• Why are the previous claims made under a paragraph called "Novelties"? For me it sounds more like "Features", unless you can clearly compare with the current state-of-the-art.

Regarding the software:

(note: as I do not use Linux, I only compiled your software and run its test cases, plus I read the documentation)

• having to build the application, to use it, is very inconvenient. When you make a release, you can add the generated packaged output (eg ade-assembly-1.0.4-bin.tgz) on GitHub, eg at https://github.com/openmainframeproject/ade/releases
  Right now, you do not have any release. Please make a new one.

• "JDK 1.7 or higher". That is not true. I tried with JDK 11, and it does not compile. Note that JDK 9 broke backward compatibility. In your case, it might be a matter of just adding the missing XML libraries that were removed from the JDK.
  Or simply specify only the JDK versions you have tested.

• in README.md: "Please see http://openmainframeproject.github.io/ade/ for documentation on ADE". Before such line, please add a "Documentation" section header, to make it clear

• I see you have a .travis.yml file, but no badge on your README.md. Add the status badge. However, notice that Travis is no longer free for open-source software. Unless you are paying for it, I would suggest to switch to either GitHub Actions or CircleCI

• you have JUnit tests, and JaCoCo configured. But your test coverage seems quite low. Eg, from ade-core/target/site/jacoco/index.html I can see there is only 7% instruction coverage. You should add some more test cases, and also possibly enable intro-module JaCoCo coverage calculation

• publish the coverage results online, and add a badge on README.md. This can be done for example with codecov.io
  For a concrete example of these badges, see one of my repositories, eg https://github.com/EMResearch/EvoMaster

Maven build fails on Travis build server. Does not fail on standard Linux instance.

This came from the TAC discussion:

A project roadmap is defined, which should address the following questions.

What use cases are possible now?
What does the next year look like in terms of additional features and use cases covered?

https://app.snyk.io/vuln/SNYK-JAVA-ORGAPACHEDERBY-30020

ADE gives the following error:

[INFO] Reactor Summary for ade 1.0.4:
[INFO] 
[INFO] ade ................................................ SUCCESS [  3.089 s]
[INFO] ade-core ........................................... SUCCESS [01:12 min]
[INFO] ade-ext ............................................ SUCCESS [ 33.735 s]
[INFO] ade-assembly ....................................... FAILURE [  0.520 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:49 min
[INFO] Finished at: 2022-06-22T10:37:34-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project ade-assembly: Could not resolve dependencies for project org.openmainframe.ade:ade-assembly:pom:1.0.4: Could not find artifact log4j:log4j:jar:2.15.0 in central (https://repo.maven.apache.org/maven2) -> [Help 1]

To reproduce:

mvn clean package

Need to create junit test for AdeUtilMain

• version
• database version
• help
• syntax of flowlayout

verify option -f which verifies content of flowlayout.xml file fails with file not found exception

how to use the rfc5434 parser in a standalone app? it fails with slf4j error or missing class

Update pom.xml file to support Java 11

Re-enable unit test using surefire

The Junit test org.openmainframe.ade.impl.dataStore.DataStoreSqlImplTest for the develop branch fails. This is caused by following code because Junit test did not mockup s_driverType.get()

 private void unlockTables() throws SQLException {
        switch (s_driverType.get()) {
            case MY_SQL:
                executeDml("unlock tables");
                break;
            case MARIADB:
                executeDml("unlock tables");
                break;
            default:
                // nothing to do for other types
        }

mvn clean package gives the following:

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.226 sec - in org.openmainframe.ade.ext.os.parser.TestInputTimeZoneManager
Running org.openmainframe.ade.ext.os.parser.TestLinuxSyslogLineParser
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

This issue is raised in reference to JOSS submission 3052.

The paper includes a paragraph on others' work on automated log anomaly detection. However, the cited articles are a conference paper and a pair of arXiv preprints. Are there any refereed articles on this topic that can be cited?

This issue is raised in reference to JOSS submission 3052.

There are numerous spelling errors, small typos, and omissions in the submitted paper. It would benefit from another editing pass by the authors.

Running the regression test using analysis_comp_test.sh fails with return code 99 - results do not compare. A review of the comparison which is stored in /tmp/compare_xxxxxxxxxx.out has determined that the message id (message key) does not match the expected value. The message key contains the correct component, but does not contain the correct sequence number.

This does not appear to change the analytics results.

Need regression test to validate that text version of model from baseline matches text version created by current version of ADE

Incorrect analysis group file created

This issue is raised in reference to JOSS submission 3052.

There's an unsubstantiated performance claim on line 48 of the manuscript. The claim should be backed up with data or struck.

Per Active stage requirements

Random comparison of interval_nn,xml files when calculating the Poisson mean (Expected text value '0.02097365841662402' but was '0.02097365841662424'). Note the difference in value is clearly insignificant - should be rounding the value before making the comparison

Message generated
Comparing /home/caffreyj/Ade_test_develop/ade-1.0.3/baseline/output/sys1.openmainframe.org/20151130/intervals/interval_48.xml to output/regtest20160907112433/continuous/sys1.openmainframe.org/20151130/intervals/interval_48.xml
** Difference Found:
Expected text value '0.02097365841662402' but was '0.02097365841662424' - comparing <poisson ...>0.02097365841662402 at /interval[1]/interval_message[7]/poisson[1]/text()[1] to <poisson ...>0.02097365841662424 at /interval[1]/interval_message[7]/poisson[1]/text()[1]

XML files are not similar

This issue is raised in reference to JOSS submission 3052.

Overall, the documentation is very helpful. However, the scoring measures used, including:

• Interval Anomaly score
• Bernoulli Score
• Best-of-two score
• Log normal score
• Poisson score
• rarity and severity scores
• critical and unique words for each message

are mentioned in the paper and in the documentation, but they aren't defined in either. Also, how are these scores combined to create the final score? This information should be in either the paper or the documentation.