yahoo / egads Goto Github PK

It does not appear the the ma.update() will do anything considering non of the tsmm model implement the update() method. Is it expected that when using this library it is always for just one off data anomaly detection? or should we implement the update() method yourself.

I would like to use this library a a backbone for some stream processing of data and update model and detect new anomalies in real time. Am i missing something about how the library should be used?

I may be mistaken, but the code at https://github.com/yahoo/egads/blob/master/src/main/java/com/yahoo/egads/models/tsmm/AutoForecastModel.java#L74 appears like the olympic model is always selected when using the automatic forecast option.

How to input data using command line and not CSV? I want be able to pass the input data through the command line so I was wondering what format that should be in? A string, or something? What delimiters to use?

Hi Security Team,

The build files indicate that this project is resolving dependencies over HTTP instead of HTTPS. Any of these artifacts could have been MITM to maliciously compromise them and infect the build artifacts that were produced. Additionally, if any of these JARs or other dependencies were compromised, any developers using these could continue to be infected past updating to fix this.

Description:

This attack leverages the build infrastructure loading dependencies over HTTP without any other sort of integrity check to allow them to be maliciously compromised.

POC code has existed since 2014 to maliciously compromise a JAR file inflight.
See:

https://max.computer/blog/how-to-take-over-the-computer-of-any-java-or-clojure-or-scala-developer/

https://github.com/mveytsman/dilettante

Source Locations :

https://github.com/yahoo/egads/blob/f14b898e3d343c2bc0318572e2c348aa9970ae74/pom.xml

  <pluginRepositories>
    <pluginRepository>
      <id>bintray-yahoo-maven</id>
      <name>bintray-plugins</name>
      <url>http://yahoo.bintray.com/maven</url>
    </pluginRepository>
  </pluginRepositories>

References :

1. https://serverfault.com/a/153065
2. https://security.stackexchange.com/a/12050
3. https://thenextweb.com/insights/2017/12/11/comcast-continues-to-inject-its-own-code-into-websites-you-visit/#
4. https://medium.com/bugbountywriteup/want-to-take-over-the-java-ecosystem-all-you-need-is-a-mitm-1fc329d898fb
5. github/codeql#2413
6. GHSA-rcj2-vvjx-87pm
7. GHSA-jwqm-c9f2-2cq3

This vulnerability has a CVSS v3.0 Base Score of 8.1/10

https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H

Hi @ALL,

I was just trying to compile with the command "mvn clean compile assembly:single" and I got:

[INFO] Scanning for projects... Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-assembly-plugin/2.2-beta-5/maven-assembly-plugin-2.2-beta-5.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-assembly-plugin/2.2-beta-5/maven-assembly-plugin-2.2-beta-5.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5: Plugin org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-assembly-plugin:jar:2.2-beta-5 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-release-plugin/2.5.2/maven-release-plugin-2.5.2.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-release-plugin/2.5.2/maven-release-plugin-2.5.2.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-release-plugin:2.5.2: Plugin org.apache.maven.plugins:maven-release-plugin:2.5.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-release-plugin:jar:2.5.2 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-deploy-plugin/2.8.2/maven-deploy-plugin-2.8.2.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-deploy-plugin/2.8.2/maven-deploy-plugin-2.8.2.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-deploy-plugin:2.8.2: Plugin org.apache.maven.plugins:maven-deploy-plugin:2.8.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-deploy-plugin:jar:2.8.2 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-compiler-plugin/3.0/maven-compiler-plugin-3.0.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-compiler-plugin/3.0/maven-compiler-plugin-3.0.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-compiler-plugin:3.0: Plugin org.apache.maven.plugins:maven-compiler-plugin:3.0 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-compiler-plugin:jar:3.0 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-clean-plugin:2.5: Plugin org.apache.maven.plugins:maven-clean-plugin:2.5 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-clean-plugin:jar:2.5 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-resources-plugin/2.6/maven-resources-plugin-2.6.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.6/maven-resources-plugin-2.6.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-resources-plugin:2.6: Plugin org.apache.maven.plugins:maven-resources-plugin:2.6 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-resources-plugin:jar:2.6 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-jar-plugin/2.4/maven-jar-plugin-2.4.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.4/maven-jar-plugin-2.4.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-jar-plugin:2.4: Plugin org.apache.maven.plugins:maven-jar-plugin:2.4 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-jar-plugin:jar:2.4 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-surefire-plugin/2.12.4/maven-surefire-plugin-2.12.4.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.12.4/maven-surefire-plugin-2.12.4.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-surefire-plugin:2.12.4: Plugin org.apache.maven.plugins:maven-surefire-plugin:2.12.4 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-surefire-plugin:jar:2.12.4 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-install-plugin/2.4/maven-install-plugin-2.4.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.4/maven-install-plugin-2.4.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-install-plugin:2.4: Plugin org.apache.maven.plugins:maven-install-plugin:2.4 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-install-plugin:jar:2.4 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-site-plugin/3.3/maven-site-plugin-3.3.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-site-plugin/3.3/maven-site-plugin-3.3.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-site-plugin:3.3: Plugin org.apache.maven.plugins:maven-site-plugin:3.3 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-site-plugin:jar:3.3 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-antrun-plugin/1.3/maven-antrun-plugin-1.3.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-antrun-plugin/1.3/maven-antrun-plugin-1.3.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-antrun-plugin:1.3: Plugin org.apache.maven.plugins:maven-antrun-plugin:1.3 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-antrun-plugin:jar:1.3 [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5: Plugin org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-assembly-plugin:jar:2.2-beta-5 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-dependency-plugin:2.8: Plugin org.apache.maven.plugins:maven-dependency-plugin:2.8 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-dependency-plugin:jar:2.8 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-release-plugin/2.5.3/maven-release-plugin-2.5.3.pom Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-release-plugin/2.5.3/maven-release-plugin-2.5.3.pom [WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-release-plugin:2.5.3: Plugin org.apache.maven.plugins:maven-release-plugin:2.5.3 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-release-plugin:jar:2.5.3 Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/apache/maven/plugins/maven-metadata.xml Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml Downloading from bintray-yahoo-maven: http://yahoo.bintray.com/maven/org/codehaus/mojo/maven-metadata.xml Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml

Obviously maven trys to download something from http://yahoo.bintray.com.
I could follow the path until http://yahoo.bintray.com/maven/org/apache/ - The subdirectory "maven" and the following directorys are missing.

Is there any other way to compile this ?

Thanks a lot !

Bye, Dirk

i installed the latest version of java JDK and maven on win10, then i successfully compiled using command mvn clean compile assembly:single, but when i type this
java -Dlog4j.configurationFile=src/test/resources/log4j2.xml -cp target/egads-*-jar-with-dependencies.jar com.yahoo.egads.Egads src/test/resources/sample_config.ini src/test/resources/sample_input.csv
an error occurs:
Error: Could not find or load main class com.yahoo.egads.Egads

Hello,
Trying to build the library using the command:

mvn clean compile assembly:single

But I got the following error:

I tried to follow with issue #36, but their solution does not work with me!
Any thought?
Thanks,

Hi,

I am trying to specify a list of time-series that are my "normal" behavior and a list or an individual time-series which I want to check for anomalies against the "normal" list - is there a way to do this under the current release?
If not, is it preferable that I work on adding another TSMM model which takes in a list of .csv files as "normal" time-series and averages them at each time unit to calculate an "expected" time-series and then compares the other time-series with this?

Thanks.

Hi,

Did you consider maybe releasing egads under different, less restrictive, license (preferably Apache)? The reason I’m asking is that right now, at least for me, egads is more like a library that I can plug into my code rather then “a tool”. And obviously, current GPL license, is problematic from this PoV.

Thanks

Hello
Is there anyone to give detailed information about System Integration ? 2.1 System Integration passage at this document ( https://s.yimg.com/ge/labs/v2/uploads/kdd2015.pdf ) there is a figure (Figure 1) . Is there anyone to explain other blue box at that figure ?
The question is how can we integrate our monitoring system. How can we use streaming data, anomaly db, config db, model db etc.

best regards
Murat

I have tried making build and i am getting error i am attaching image of error and also my version information in the same picture. Can you please tell me what could be the problem.

it is not clear from the code how to use the engine. specifically - how to generate the expected series.

in the AnomalyDetectionModel there is the following method:

// detect anomalies.
public Anomaly.IntervalSequence detect(
        TimeSeries.DataSequence observedSeries,
        TimeSeries.DataSequence expectedSeries) throws Exception;

I'm not sure how the expectedSeries is produced?

in the unit tests it is loaded from a file: src/test/resources/model_output_" + refWindows[w] + "_" + drops[d] + ".csv" which is also - not understood.

any help is appreciated.

I'm considering this library for a project where I periodically fetch metric data and want to test each new data point against previous values to see if the latest value is an anomaly.

However, the configuration files and tests seem to use static data sets (CSV files). The entire data set is loaded upfront, and then the analysis is performed. For example:
https://github.com/yahoo/egads/blob/master/src/test/java/com/yahoo/egads/TestOlympicModel.java#L85

On the other hand, it appears that a TimeSeries object can be instantiated and values added without using a CSV. https://github.com/yahoo/egads/blob/master/src/test/java/com/yahoo/egads/TestTimeSeries.java#L23

I'm wondering if it makes sense to instantiate a Time Series object, and then recreate the model and perform the analysis each time I update it with a new data point, or if there is a preferred method for handling streaming data.

Thank you.

I am reading the source code, but can't understand DBScanModel, the 'detect' method which judge a point whether or not an anomly in DBScanModel.java, it just judges observedValue != expectedValue, so I am confuse if it is useful for using DBScanClusterer to cluster the points.

if (observedSeries.get(p.getId()).value != expectedSeries.get(p.getId()).value && ((((unixTime - observedSeries.get(p.getId()).time) / 3600) < maxHrsAgo) || (maxHrsAgo == 0 && p.getId() == (n - 1)))) { output.add(new Interval(observedSeries.get(p.getId()).time, p.getId(), errors, thresholdErrors, observedSeries.get(p.getId()).value, expectedSeries.get(p.getId()).value)); }

This property ought to be public so that consumers of this class can inspect which model is ideal.

https://github.com/yahoo/egads/blob/master/src/main/java/com/yahoo/egads/models/tsmm/AutoForecastModel.java#L36

The DoubleExponentialSmoothingModel time-series model specifies the name as DoubleExponentialSmothingModel (missing the second 'o' in smoothing).

When I change the TS_MODEL to AutoForecastModel in config and run it on my data, it got stack overflow like this:

Exception in thread "main" java.lang.StackOverflowError
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.forecast(TripleExponentialSmoothingModel.java:673)
    at net.sourceforge.openforecast.models.AbstractTimeBasedModel.initForecastValue(AbstractTimeBasedModel.java:261)
    at net.sourceforge.openforecast.models.AbstractTimeBasedModel.getForecastValue(AbstractTimeBasedModel.java:361)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.getSeasonalIndex(TripleExponentialSmoothingModel.java:817)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.getBase(TripleExponentialSmoothingModel.java:734)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.forecast(TripleExponentialSmoothingModel.java:673)
    at net.sourceforge.openforecast.models.AbstractTimeBasedModel.initForecastValue(AbstractTimeBasedModel.java:261)
    at net.sourceforge.openforecast.models.AbstractTimeBasedModel.getForecastValue(AbstractTimeBasedModel.java:361)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.getSeasonalIndex(TripleExponentialSmoothingModel.java:817)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.getBase(TripleExponentialSmoothingModel.java:734)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.forecast(TripleExponentialSmoothingModel.java:673)
    at net.sourceforge.openforecast.models.AbstractTimeBasedModel.initForecastValue(AbstractTimeBasedModel.java:261)
    at net.sourceforge.openforecast.models.AbstractTimeBasedModel.getForecastValue(AbstractTimeBasedModel.java:361)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.getSeasonalIndex(TripleExponentialSmoothingModel.java:817)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.getBase(TripleExponentialSmoothingModel.java:734)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.forecast(TripleExponentialSmoothingModel.java:673)
    at net.sourceforge.openforecast.models.AbstractTimeBasedModel.initForecastValue(AbstractTimeBasedModel.java:261)
    at net.sourceforge.openforecast.models.AbstractTimeBasedModel.getForecastValue(AbstractTimeBasedModel.java:361)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.getSeasonalIndex(TripleExponentialSmoothingModel.java:817)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.getBase(TripleExponentialSmoothingModel.java:734)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.forecast(TripleExponentialSmoothingModel.java:673)
    at net.sourceforge.openforecast.models.AbstractTimeBasedModel.initForecastValue(AbstractTimeBasedModel.java:261)
    at net.sourceforge.openforecast.models.AbstractTimeBasedModel.getForecastValue(AbstractTimeBasedModel.java:361)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.getSeasonalIndex(TripleExponentialSmoothingModel.java:817)

.... 

at net.sourceforge.openforecast.models.AbstractTimeBasedModel.initForecastValue(AbstractTimeBasedModel.java:261)
    at net.sourceforge.openforecast.models.AbstractTimeBasedModel.getForecastValue(AbstractTimeBasedModel.java:361)
    at net.sourceforge.openforecast.models.TripleExponentialSmoothingModel.getSeasonalIndex(TripleExponentialSmoothingModel.java:817)

And it works correctly on other models.

I can't find any documentation about how to configure each one of the models (AD or TSMM)
Is there any detailed list of parameters and how to configure them?
Thanks!

Can this be used with online/streamed data?

Public static nonfinal fields are not conducive to a concurrent, enterprise-ready models. Refactoring the code to remove this class will bring this project one step closer to production-level quality.

I cloned the latest repository (main branch) and ran the maven build command as specified by the README. I get various build errors that say that Maven is using Java 1.3 and I should use Java 1.5 or greater.

Apache Maven 3.0.5
Maven home: /usr/share/maven
Java version: 1.8.0_60, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.16.0-31-generic", arch: "amd64", family: "unix"

I see 4.2 Filtering in paper -- Generic and Scalable Framework for Automated Time-series Anomaly Detection, but i can not find filter module in the code. So, is that i missed it?

I have tried to simply compile the code (on ubuntu 14.04) but fail :

Steps performed till now :

1. git clone
2. export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64
3. mvn clean compile assembly:single

This yields the following error:

[INFO] Compilation failure
/home/kiran/toolbox/egads-yahoo-AD/egads/src/main/java/com/yahoo/egads/models/adm/AdaptiveKernelDensityChangePointDetector.java:[172,82] cannot find symbol
symbol: method join(java.lang.String,java.lang.String)
location: class java.lang.String

I am not familiar with java code and do not understand what could be the problem.

I saw that the paper https://labs.yahoo.com/_c/uploads/KDD2015.pdf has mentioned about three module: the time-series modeling module (TMM), the anomaly detection module (ADM) and the alerting module (AM). But I only found TMM (com.yahoo.egads.models.tsmm) and ADM (com.yahoo.egads.models.adm) in the package. Where is the AM?

Is the threshold selection used only in Outliers or in all three classes of anomalies？

The program can potentially fail to release a system resource.

Hi,

I try to detect anomaly detection in time-series with last sales and forecast sales for next x weeks.

The time-series have seasonality and holidays outliers.

I have 3 years of data and sales is grouped by week.

What the recommended AD_MODEL for this case?

I would like to use this software as part of a backend to detect anomalies on data collected using the sensors of the phone. Apparently, this software package still does not support online anomaly detection. Anyway, what would be the advisable approach to use EGADS in the Android setting? I would like to detect anomalies in the GPS data, which is stored in a CSV file. Any suggestions?

i installed the latest version of java JDK and maven on win10, then i successfully compiled using command mvn clean compile assembly:single, but when i type this
java -Dlog4j.configurationFile=src/test/resources/log4j2.xml -cp target/egads-*-jar-with-dependencies.jar com.yahoo.egads.Egads src/test/resources/sample_config.ini src/test/resources/sample_input.csv
an error occurs:
Error: Could not find or load main class com.yahoo.egads.Egads
Caused by: java.lang.ClassNotFoundException: com.yahoo.egads.Egads

Instead of hosting the OpenForecast jar on github, we can use the maven-external-dependency plugin to download the jar and use it as an artifact. This would be more preferable than distributing this jar through source code management.

https://code.google.com/p/maven-external-dependency-plugin/

I cant compile the code (java 1.7, ubuntu 14.04) due to a generics error (unchecked call) that, I think, was only a warning in previous java versions.

Is there a way to resolve this issue?

Using the automatic forecasting option for a small dataset always results in an error from the TripleExponentialSmoothing model which states it needs 2 years of data, which is fair enough.

However, I don't think this should kill the automatic forecasting. Maybe errors from models could be caught, and only the models which successfully built could be assessed?