fracpete / multisearch-weka-package

Weka package for parameter optimization, similar to GridSearch, but with arbitrary number of parameters.

multisearch-weka-package

Weka package for parameter optimization, similar to GridSearch. Can be used for optimizing an arbitrary number of parameters, in contrast to GridSearch which always requires you to optimize two parameters. However, it does not offer automatic search space extensions like GridSearch.

Parameters

• weka.core.setupgenerator.MathParameter

  uses mathematical expression to compute the (numeric) parameters

• weka.core.setupgenerator.ListParameter

  list of values; default is blank-separated, but a custom delimiter can be supplied

• weka.core.setupgenerator.MLPLayersParameter

  generates hidden layer definitions for the MultiLayerPerceptron classifier (contributed by Jan van Rijn)

• weka.core.setupgenerator.ParameterGroup

  allows grouping of dependent parameters, e.g., setting on group sets the kernel of SMO to RFBKernel and explores the gamma option, another group sets the kernel to PolyKernel and explores the exponent option.

Note: array elements, e.g., the filters inside a weka.filters.MultiFilter can be accessed using [n] with n being the 0-based index. E.g., if the third filter inside a MultiFilter is a PLSFilter, then its numComponents property can be accessed with filters[2].numComponents.

Supported parameter types

• char, string
• float, double
• int, long
• boolean
• weka.core.SelectedTag
• Java classname (and possible options for classes implementing weka.core.OptionHandler)

Search space exploration

The search space of setups can be explored with different strategies, derived from weka.classifiers.meta.multisearch.AbstractSearch. The following strategies are available:

• weka.classifiers.meta.multisearch.DefaultSearch

  Exhaustive search of parameter space

• weka.classifiers.meta.multisearch.RandomSearch

  Random search of parameter space (contributed by Jan van Rijn)

Example

With the following classifier setup:

weka.classifiers.meta.FilteredClassifier
|
+- weka.filters.supervised.attribute.PLSFilter
|
+- weka.classifiers.functions.LinearRegression

You can explore the filter's PLS components and classifier's ridge parameters by referencing these properties as follows (the MultiSearch's classifier is used as the base for the property paths):

• components: filter.numComponents
• ridge: classifier.ridge

For more examples, please see the following repository:

https://github.com/fracpete/multisearch-weka-package-examples

Releases

Click on one of the following links to download the corresponding Weka package:

• 2021.2.17
• 2020.2.17
• 2019.10.4
• 2019.9.30
• old releases: 2018.9.2, 2018.8.16, 2018.8.11, 2017.10.1, 2017.3.28, 2016.6.6, 2016.5.31, 2016.5.25, 2016.5.15, 2016.4.30, 2016.4.25, 2016.4.14, 2016.1.30, 2016.1.15, 2016.1.14, 2016.1.13, 2015.10.15, 2015.9.2, 2014.12.10

How to use packages

For more information on how to install the package, see:

https://waikato.github.io/weka-wiki/packages/manager/

Maven

Add the following dependency in your pom.xml to include the package:

    <dependency>
      <groupId>com.github.fracpete</groupId>
      <artifactId>multisearch-weka-package</artifactId>
      <version>2021.2.17</version>
      <type>jar</type>
      <exclusions>
        <exclusion>
          <groupId>nz.ac.waikato.cms.weka</groupId>
          <artifactId>weka-dev</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

multisearch-weka-package's Issues

Add weighted AUC as measure

Similar to GridSearch, add weighted AUC as measure available for optimization.

In weka Experimenter, MultiSearch raises 'java.lang.NullPointerException'

Weka version: 3.7.12
multisearch version: 2015.9.2

Error msg:

Running experiment: Runs from: 1 to: 1
Datasets: D:\Hans\CNTT\151001데이터마이닝\weka\06 기념일추가 징검다리휴일추가 월,주차추가\06.arff
Custom property iterator: on
Custom property path:
1  weka.experiment.RandomSplitResultProducer::splitEvaluator RegressionSplitEvaluator: weka.classifiers.rules.ZeroR (version 48055541465867954)
Custom property name:classifier
Custom property values:
 1 weka.classifiers.meta.MultiSearch No search performed yet.
ResultProducer: RandomSplitResultProducer: -P 80.0 -R -W weka.experiment.RegressionSplitEvaluator --: <null Instances>
ResultListener: weka.experiment.CSVResultListener@71c410e8

Writing experiment copy
Reading experiment copy
Made experiment copy
java.lang.NullPointerException
        weka.classifiers.meta.MultiSearch.enumerateMeasures(MultiSearch.java:926)
        weka.experiment.Experiment.determineAdditionalResultMeasures(Experiment.java:481)
        weka.experiment.Experiment.initialize(Experiment.java:429)
        weka.gui.experiment.RunPanel$ExperimentRunner.run(RunPanel.java:155)

        at weka.classifiers.meta.MultiSearch.enumerateMeasures(MultiSearch.java:926)
        at weka.experiment.Experiment.determineAdditionalResultMeasures(Experiment.java:481)
        at weka.experiment.Experiment.initialize(Experiment.java:429)
        at weka.gui.experiment.RunPanel$ExperimentRunner.run(RunPanel.java:155)
null
Done...

Support for one optimized parameter

If I try to use the package (via the Python wrapper) with one parameter:

    mparam = MathParameter()
    mparam.prop = "classifier.minBucketSize"
    mparam.minimum = 1
    mparam.maximum = 10
    mparam.step = 1
    mparam.expression = "I"     
    multi.parameters = [mparam]      
    multi.classifier = Classifier(classname"weka.classifiers.rules.OneR")      

I get the the following error:

            Exception in thread "Thread-0" java.lang.IllegalStateException: All points were already cached - abnormal state!
            at weka.classifiers.meta.MultiSearch.determineBestInSpace(MultiSearch.java:1336)
            at weka.classifiers.meta.MultiSearch.findBest(MultiSearch.java:1428)
            at weka.classifiers.meta.MultiSearch.buildClassifier(MultiSearch.java:1481)

If indeed the error I am getting is due to the number of parameters being 1, I am not sure if one parameter optimization is not supported by design, or this is a bug.

In case 2+ parameters are required, I think there could be some sort of illegal configuration error thrown earlier.

Parameter/SpaceDimension refactoring

To ease the current tight coupling of classes:

• make SpaceDimension an interface
• each parameter class defines a SpaceDimension class, which also handles the subdimension call
• parameter returns a custom SpaceDimension object via spaceDimensions()

how to get the information about the best selected values of tuned parameters?

Hello there,

i am using MultiSearch package so as to tune a Neural Network. I am using two times the ListParameters option and i am placing there the values of corresponding parameters so as to select the best ones. However, running Weka through API, i do not obtain the information about the selected combination of values but only the Accuracy value and the Confusion matrix.
Does any option exist about this request?

Thanks in advance.

Optimizing metrics on a specific class

Hi, I'm wondering if it is possible to optimize a metric (eg. F1) for a specific class. From what I understand, by default we're optimizing for the mean F1 over all classes.

Thanks for your help, this has been a very useful package!

Output best parameters in "Summary" column in experement result.

While I can review selected parameters from Explorer or debug output from console, I can't find it from Experiment result. This is annoying because I need to re-run the classifier in Explorer to find out the best parameters.

There is "Summary" column in experiment result(arff, xlsx, csv, ...) and I expected that I can get experiment results from there. I'm not sure, but I recall that other algorithms like GridSearch outputs selected parameters there.

For example, this "Summary" result is from Experimenter.

And this is from Excel-formated result.

LibSVM weight parameter requires a blank-separated list.

It is not intuitive (or possible?) to tune parameters that require blank-separated lists as input (maybe allow specifying list separators?). Also, the escaping situation is godawful.

java weka.classifiers.meta.MultiSearch -E PRC -t train.arff -W weka.classifiers.functions.LibSVM -search 'weka.core.setupgenerator.MathParameter -property classifier.cost -min -3 -max 2 -step 1.0 -base 10.0 -expression "pow(BASE, I)"' -search 'weka.core.setupgenerator.MathParameter -property classifier.gamma -min -2 -max 2 -step 1.0 -base 10.0 -expression "pow(BASE, I)"' -search 'weka.core.setupgenerator.ListParameter -property classifier.weights -list "\"1 100\" \"1 10\" \"1 1\""' -d LibSVM.model -log-file LibSVM-TUNED.log

Zero Weights processed. Default weights will be used

Which doesn't sound like what I want. Can't really figure out how I should quote/not quote things to make it work...

Weka exception when running with ADTree

java weka.classifiers.meta.MultiSearch -E PRC -t train.arff -W "weka.Run weka.classifiers.trees.ADTree" -search 'weka.core.setupgenerator.MathParameter -property classifier.numOfBoostingIterations -min 5 -max 25 -step 5 -expression I' -d ADTree.model -log-file ADTree-TUNED.log

Weka exception: Can't find class called: weka.Run weka.classifiers.trees.ADTree ...

Actually this might be a weka issue, since I run into a similar issue just trying to run the ADTree class in weka 3.8 (I'm using weka 3.8.0), but it blocks me from tuning with MultiSearch.

Add new test method using hold-out data set

Currently, cross-validation is the only way to evaluate parameter using multisearch. But there are some situations that data order should be preserved. Time-related reserches like 'stock market prediction' or 'sale prediction' are such cases in my opinion. We should not use future data to estimate past, after all.

Can I request for a new feature to use hold-out data set for evaluation?

BTW, I'm not good at writing English. Sorry if I missed some manners.

In weka explorer, MultiSearch raises 'java.lang.NoClassDefFoundError'

Weka version: 3.7.13
multisearch version: 2015.9.2 (But displayed as 2014.12.10 in weka package manager. Maybe another issue?)

I set class attribute, but still raises error.

Error msg:

java.lang.NoClassDefFoundError: weka/core/MathematicalExpression
        weka.core.SetupGenerator.evaluate(SetupGenerator.java:380)
        weka.classifiers.meta.multisearch.DefaultEvaluationTask.doRun(DefaultEvaluationTask.java:68)
        weka.classifiers.meta.multisearch.AbstractEvaluationTask.run(AbstractEvaluationTask.java:88)
        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        java.lang.Thread.run(Unknown Source)

        at weka.core.SetupGenerator.evaluate(SetupGenerator.java:380)
        at weka.classifiers.meta.multisearch.DefaultEvaluationTask.doRun(DefaultEvaluationTask.java:68)
        at weka.classifiers.meta.multisearch.AbstractEvaluationTask.run(AbstractEvaluationTask.java:88)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

Exception when tuning weights with libsvm

Hi, I'm running version 2016.5.25, and am hitting the following exception:

+ java weka.classifiers.meta.MultiSearch -E PRC -t train.arff -W weka.classifiers.functions.LibSVM -d LibSVM.model -log-file LibSVM.log -search 'weka.core.setupgenerator.MathParameter -property classifier.cost -min -3 -max 2 -step 1.0 -base 10.0 -expression "pow(BASE, I)"' -search 'weka.core.setupgenerator.MathParameter -property classifier.gamma -min -2 -max 2 -step 1.0 -base 10.0 -expression "pow(BASE, I)"' -search 'weka.core.setupgenerator.ListParameter -property classifier.weights -list "\"1 100\" \"1 10\" \"1 1\""' Zero Weights processed. Default weights will be used java.beans.IntrospectionException: Method not found: isClassifier at java.beans.PropertyDescriptor.<init>(PropertyDescriptor.java:107) at java.beans.PropertyDescriptor.<init>(PropertyDescriptor.java:71) at weka.core.PropertyPath.find(PropertyPath.java:386) at weka.core.SetupGenerator.setup(SetupGenerator.java:592) at weka.classifiers.meta.multisearch.DefaultEvaluationTask.doRun(DefaultEvaluationTask.java:83) at weka.classifiers.meta.multisearch.AbstractEvaluationTask.run(AbstractEvaluationTask.java:99) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) java.beans.IntrospectionException: Method not found: isClassifier at java.beans.PropertyDescriptor.<init>(PropertyDescriptor.java:107) at java.beans.PropertyDescriptor.<init>(PropertyDescriptor.java:71) at weka.core.PropertyPath.find(PropertyPath.java:386) at weka.core.SetupGenerator.setup(SetupGenerator.java:592) at weka.classifiers.meta.multisearch.DefaultEvaluationTask.doRun(DefaultEvaluationTask.java:83) at weka.classifiers.meta.multisearch.AbstractEvaluationTask.run(AbstractEvaluationTask.java:99) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) java.beans.IntrospectionException: Method not found: isClassifier at java.beans.PropertyDescriptor.<init>(PropertyDescriptor.java:107) at java.beans.PropertyDescriptor.<init>(PropertyDescriptor.java:71) at weka.core.PropertyPath.find(PropertyPath.java:386) at weka.core.SetupGenerator.setup(SetupGenerator.java:592) at weka.classifiers.meta.multisearch.DefaultEvaluationTask.doRun(DefaultEvaluationTask.java:83) at weka.classifiers.meta.multisearch.AbstractEvaluationTask.run(AbstractEvaluationTask.java:99) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ....* optimization finished, #iter = 4808 nu = 0.23357145684034458 obj = -554.5871835474204, rho = -0.805166553228126

The issue appears to be with tuning the weight parameters. Perhaps I am not escaping the list correctly?

Incorrect debug output when evaluating using a test set instead of a k-fold CV

When specifying a test set, the debug output says that 2-fold CV is used:

...
=== Initial space - Start ===
Determining best values with 2-fold CV in space:
...

I checked the code and the corresponding if statement generating this output does not check whether a test set has been specified.

Add code samples

This package is one of the most useful things there is for Weka.

However, I'm having to do quite a bit of guesswork on how to use it properly in code, because there are 0 examples.

I'm going on mostly based on the little info there is at https://weka.wikispaces.com/Optimizing+parameters#MultiSearch and my own trial and error.

Be great if there were code samples loading iris and doing some parameters search for a decision tree or some simple classifier that takes multiple parameters.

Weird threading issue

I'm using the package and I'm getting weird threading problems. Basically my main program never finishes because some threads are left hanging.

I'm guessing it's because of this package, because it's the only place I'm using threads.

I do not get any errors, the code simply hangs with a print statement I put before the last line of main.
The weird thing is that if I run the code for few iterations I do not get the hanging problem. Basically it is a learning method that has to do parameter optimization on every iteration. I'm using 6 threads to do the search. When I have lots of iterations (say more than 100) I get the hanging.

A stack after the hanging point is as follows:

175386 [main] DEBUG root  - generate returned. ready to finish main
^\2016-05-09 13:05:41
Full thread dump OpenJDK 64-Bit Server VM (24.95-b01 mixed mode):

"DestroyJavaVM" prio=10 tid=0x00007f57b000b000 nid=0x2263 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"pool-15-thread-6" prio=10 tid=0x00007f57b0204800 nid=0x22f2 waiting on condition [0x00007f57a3dc0000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007883a4de0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

"pool-15-thread-5" prio=10 tid=0x00007f57b0203800 nid=0x22f1 waiting on condition [0x00007f57a39bc000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007883a4de0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

"pool-15-thread-4" prio=10 tid=0x00007f57b041a800 nid=0x22ef waiting on condition [0x00007f57a3abd000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007883a4de0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

"pool-15-thread-3" prio=10 tid=0x00007f57b0410000 nid=0x22ee waiting on condition [0x00007f57a3cbf000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007883a4de0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

"pool-15-thread-2" prio=10 tid=0x00007f57b0414000 nid=0x22ed waiting on condition [0x00007f57a3bbe000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007883a4de0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

"pool-15-thread-1" prio=10 tid=0x00007f57b0202800 nid=0x22ec waiting on condition [0x00007f57a81be000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007883a4de0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

"Service Thread" daemon prio=10 tid=0x00007f57b00b3000 nid=0x2270 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007f57b00b1000 nid=0x226f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007f57b00ae000 nid=0x226e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f57b00ac000 nid=0x226d waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f57b0081000 nid=0x226c in Object.wait() [0x00007f57aa0d1000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000785661ad8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
    - locked <0x0000000785661ad8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" daemon prio=10 tid=0x00007f57b007f000 nid=0x226b in Object.wait() [0x00007f57aa1d2000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007856626d0> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:503)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
    - locked <0x00000007856626d0> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x00007f57b007a800 nid=0x226a runnable 

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f57b0020800 nid=0x2264 runnable 

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f57b0022800 nid=0x2265 runnable 

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f57b0024800 nid=0x2266 runnable 

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f57b0026000 nid=0x2267 runnable 

"GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f57b0028000 nid=0x2268 runnable 

"GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f57b002a000 nid=0x2269 runnable 

"VM Periodic Task Thread" prio=10 tid=0x00007f57b00be000 nid=0x2271 waiting on condition 

JNI global references: 216

Heap
 PSYoungGen      total 679936K, used 209296K [0x00000007d6680000, 0x0000000800000000, 0x0000000800000000)
  eden space 678400K, 30% used [0x00000007d6680000,0x00000007e3244178,0x00000007ffd00000)
  from space 1536K, 41% used [0x00000007ffd00000,0x00000007ffda0000,0x00000007ffe80000)
  to   space 1536K, 0% used [0x00000007ffe80000,0x00000007ffe80000,0x0000000800000000)
 ParOldGen       total 147456K, used 82979K [0x0000000783400000, 0x000000078c400000, 0x00000007d6680000)
  object space 147456K, 56% used [0x0000000783400000,0x0000000788508d78,0x000000078c400000)
 PSPermGen       total 22528K, used 11721K [0x0000000778e00000, 0x000000077a400000, 0x0000000783400000)
  object space 22528K, 52% used [0x0000000778e00000,0x0000000779972420,0x000000077a400000)

Any idea if the package could have problems with finishing the optimizer threads?

What is the name of this package?

Hello, I am trying to install the multisearch pacakge from package manager but seems I can not find the package. Is it available now and what the name is it? Following is the package list I found -

Installed   Repository  Loaded  Package
=========   ==========  ======  =======
-----       1.2.1       No  AnDE: Averaged N-Dependence Estimators (includes A1DE and A2DE)
-----       1.0.0       No  ArabicStemmers_LightStemmers: Arabic Stemmer / Light Stemmer 
-----       1.0         No  CAAR: Context Aware Case-Based Regression Learner
-----       1.0.1       No  CHIRP: CHIRP: A new classifier based on Composite Hypercubes on Iterated Random Projections
-----       1.0.1       No  CLOPE: CLOPE: a fast and effective clustering algorithm for transactional data
-----       1.0.0       No  CVAttributeEval: An Variation degree Algorithm to explore the space of attributes.
-----       1.0.1       No  DMNBtext: Class for building and using a Discriminative Multinomial Naive Bayes classifier
-----       1.0.3       No  DTNB: Class for building and using a decision table/naive Bayes hybrid classifier.
-----       1.0.1       No  DilcaDistance: Learning distance measure for categorical data
-----       1.0.0       No  DistributionBasedBalance: Distribution-based balancing of datasets
-----       1.0         No  EAR4: Case-Based Regression Learner
-----       1.0.1       No  EMImputation: Replaces missing numeric values using Expectation Maximization with a multivariate normal model.
-----       1.0.0       No  EvolutionarySearch: An Evolutionary Algorithm (EA) to explore the space of attributes.
-----       1.0.0       No  GPAttributeGeneration: Genetic Programming Attribute Generation
-----       1.0.1       No  IBkLG: Log and Gaussian kernel for K-NN
-----       1.0.0       No  IWSS: Incremental Wrapper Subset Selection
-----       1.0.0       No  IWSSembeddedNB: Incremental Wrapper Subset Selection with embedded NB classifier
-----       3.0         No  J48Consolidated: Class for generating a pruned or unpruned C45 consolidated tree
-----       1.0.3       No  J48graft: Class for generating a grafted (pruned or unpruned) C4.5 decision tree
-----       1.0.0       No  JDBCDriversDummyPackage: Dummy package that provides a place to drop JDBC driver jar files so that they get loaded by the system.
-----       1.0.1       No  LVQ: Cluster data using the Learning Vector Quantization algorithm.
-----       1.9.5       No  LibLINEAR: A wrapper class for the liblinear classifier
-----       1.0.6       No  LibSVM: A wrapper class for the libsvm tools
-----       1.0.0       No  MODLEM: MODLEM rule algorithm
-----       1.0.0       No  MTreeClusterer: MTree Clusterer
-----       1.0.1       No  NNge: Nearest-neighbor-like algorithm using non-nested generalized exemplars (which are hyperrectangles that can be viewed as if-then rules)
-----       0.9.1       No  OpenmlWeka: Openml Weka
-----       1.0.0       No  PCP: Parallel Coordinates Plot
-----       1.2.0       No  PSOSearch: An implementation of the Particle Swarm Optimization (PSO) algorithm to explore the space of attributes.
-----       1.0.8       No  RBFNetwork: Classes that implement radial basis function networks.
-----       1.1.6       No  RPlugin: Execute R Scripts
-----       1.1.0       No  RerankingSearch: Meta-Search algorithm which performs a Hybrid feature selection based on re-ranking
-----       1.0.3       No  SMOTE: Resamples a dataset by applying the Synthetic Minority Oversampling TEchnique (SMOTE).
-----       1.0.1       No  SPegasos: Implements the stochastic variant of the Pegasos (Primal Estimated sub-GrAdient SOlver for SVM) method of Shalev-Shwartz et al. (2007).
-----       1.0.1       No  SSF: Simplified Silhouette Filter
-----       1.0.1       No  SVMAttributeEval: Evaluates the worth of an attribute by using an SVM classifier.
-----       1.0.3       No  SelfOrganizingMap: Cluster data using the Kohonen's Self-Organizing Map algorithm.
-----       1.4.4       No  SparseGenerativeModel: Sparse Generative Model
-----       1.0.0       No  StudentFilters: Student Filters
-----       1.0.6       No  TPP: Targeted Projection Pursuit
-----       1.0.5       No  WekaExcel: WEKA MS Excel loader/saver
-----       1.0.4       No  WekaODF: WEKA ODF loader/saver
-----       1.0.4       No  XMeans: Cluster data using the X-means algorithm.
-----       1.0.6       No  alternatingDecisionTrees: Binary-class alternating decision trees and multi-class alternating decision trees.
-----       1.0.0       No  alternatingModelTrees: Alternating Model Trees
-----       1.0.1       No  anonymizationPackage: A Filter to apply k-anonymization and l-diversity
-----       1.0.1       No  associationRulesVisualizer: A visualization component for displaying association rules that uses a modified version of the Association Rules Viewer from DESS IAGL of Lille.
-----       1.0.7       No  attributeSelectionSearchMethods: Four search methods for attribute selection: ExhaustiveSearch, GeneticSearch, RandomSearch and RankSearch.
-----       1.2.1       No  averagedOneDependenceEstimators: Family of methods for learning ensembles of naive Bayes-like classifiers (with relaxed independence assumption) - includes AODE, WAODE and AODEsr
-----       1.0.0       No  baggedLocalOutlierFactor: Filter implementing the Bagged LOF outlier/anomaly detection algorithm.
-----       1.0.2       No  bayesianLogisticRegression: Implements Bayesian Logistic Regression for both Gaussian and Laplace Priors
-----       1.0.4       No  bestFirstTree: Class for building a best-first decision tree classifier.
-----       1.0.3       No  cascadeKMeans: k-means clustering with automatic selection of k
-----       1.0.1       No  cassandraConverters: Loader and saver for the cassandra NoSQL database
-----       1.0.4       No  chiSquaredAttributeEval: Attribute evaluator that evaluates the worth of an attribute by computing the value of the chi-squared statistic with respect to the class.
-----       1.0.1       No  citationKNN: Modified version of the Citation kNN multi instance classifier
-----       1.0.3       No  classAssociationRules: Class association rules algorithms (including an implementation of the CBA algorithm).
-----       1.0.3       No  classificationViaClustering: A simple meta-classifier that uses a clusterer for classification.
-----       1.0.1       No  classificationViaRegression: Class for doing classification using regression methods.
-----       1.0.5       No  classifierBasedAttributeSelection: A subset evaluator and an attribute evaluator for evaluating the merit of subsets and single attributes respectively using a classifier.
-----       1.0.1       No  classifierErrors: A visualization component for displaying errors from numeric schemes using the JMathTools library.
-----       1.0.1       No  clojureClassifier: Wrapper classifiers for classifiers implemented in the Clojure programming language
-----       1.0.3       No  complementNaiveBayes: Class for building and using a Complement class Naive Bayes classifier.
-----       1.0.4       No  conjunctiveRule: This class implements a single conjunctive rule learner that can predict for numeric and nominal class labels.
-----       1.0.4       No  consistencySubsetEval: Evaluates the worth of a subset of attributes by the level of consistency in the class values when the training instances are projected onto the subset of attributes.
-----       1.0.3       No  costSensitiveAttributeSelection: Two meta attribute selection evaluators (one attribute-based and the other subset-based) for performing cost-sensitive attribute selection.
-----       1.0.3       No  dagging: This meta classifier creates a number of disjoint, stratified folds out of the data and feeds each chunk of data to a copy of the supplied base classifier.
-----       1.0.2       No  decorate: DECORATE is a meta-learner for building diverse ensembles of classifiers by using specially constructed artificial training examples
-----       1.0.3       No  denormalize: An instance filter that collapses instances with a common grouping ID value into a single instance.
-----       1.0.0       No  discriminantAnalysis: Code for discriminant analysis
-----       1.0.12      No  distributedWekaBase: Generic configuration classes and distributed map/reduce type tasks for Weka
-----       1.0.14      No  distributedWekaHadoop: Hadoop wrappers for Weka
-----       1.0.0       No  distributedWekaSpark: Spark wrappers for Weka
-----       1.0.5       No  ensembleLibrary: Manages a libary of ensemble classifiers
-----       1.0.3       No  ensemblesOfNestedDichotomies: A meta classifier for handling multi-class datasets with 2-class classifiers by building an ensemble of nested dichotomies.
-----       1.0.1       No  extraTrees: Package for generating a single Extra-Tree
-----       1.0.1       No  fastCorrBasedFS: Feature selection method based on correlation measureand relevance and redundancy analysis
-----       1.0.1       No  filteredAttributeSelection: Two meta attribute selection evaluators (one attribute-based and the other subset-based) for filtering data before performing attribute selection.
-----       1.0.3       No  functionalTrees: Classifier for learning Functional Trees
-----       1.0.1       No  fuzzyLaticeReasoning: The Fuzzy Lattice Reasoning Classifier uses the notion of Fuzzy Lattices for creating a Reasoning Environment
-----       1.0.2       No  fuzzyUnorderedRuleInduction: Fuzzy Unordered Rule Induction Algorithm
-----       1.0.1       No  gaussianProcesses: Implements Gaussian Processes for regression without hyperparameter-tuning.
-----       1.0.1       No  generalizedSequentialPatterns: Class implementing a GSP algorithm for discovering sequential patterns in a sequential data set
-----       1.0.1       No  grading: Implements Grading. The base classifiers are "graded".
-----       1.0.0       No  graphgram: GraphGram - Visualization for Clusterings
-----       1.0.9       No  gridSearch: Performs a grid search of parameter pairs for the a classifier.
-----       1.0.1       No  hiddenNaiveBayes: Contructs Hidden Naive Bayes classification model with high classification accuracy and AUC
-----       1.0.0       No  hiveJDBC: A package containing the JDBC driver and dependencies for the Apache Hive database, along with a DatabaseUtils.props file for use with Weka.
-----       1.0.5       No  hotSpot: HotSpot learns a set of rules (displayed in a tree-like structure) that maximize/minimize a target variable/value of interest.
-----       1.0.1       No  hyperPipes: Class implementing a HyperPipe classifier.
-----       1.0.0       No  imageFilters: A package that contains filters to process image files.
-----       1.0.1       No  isolationForest: Class for building and using a classifier built on the Isolation Forest anomaly detection algorithm.
-----       1.0.1       No  isotonicRegression: Learns an isotonic regression model.
-----       1.0.2       No  jfreechartOffscreenRenderer: Offscreen chart renderer plugin for the Knowledge Flow that uses JFreeChart
-----       1.0.0       No  jsonFieldExtractor: Extract fields from repeating JSON structures.
-----       1.0.0       No  kernelLogisticRegression: A package that contains a class to train a two-class kernel logistic regression model.
-----       1.0.9       No  kfGroovy: A Knowledge Flow plugin that provides a Knowledge Flow step that wraps around a Groovy script.
-----       1.0.3       No  kfKettle: A Knowledge Flow plugin that serves as a data source for data coming from the Kettle ETL tool.
-----       1.0.2       No  kfPMMLClassifierScoring: A Knowledge Flow plugin that provides a Knowledge Flow step for scoring test sets or instance streams using a PMML classifier.
-----       1.0.2       No  latentSemanticAnalysis: Performs latent semantic analysis and transformation of the data
-----       1.0.0       No  lazyAssociativeClassifier: Lazy Associative Classifier
-----       1.0.1       No  lazyBayesianRules: Lazy Bayesian Rules Classifier
-----       1.0.1       No  leastMedSquared: Implements a least median squared linear regression utilizing the existing weka LinearRegression class to form predictions.
-----       1.0.1       No  levenshteinEditDistance: Computes the Levenshtein edit distance between two strings
-----       1.0.1       No  linearForwardSelection: Extension of BestFirst that takes a restricted number of k attributes into account.
-----       1.0.4       No  localOutlierFactor: Filter implementing the Local Outlier Factor (LOF) outlier/anomaly detection algorithm.
-----       1.0.2       No  massiveOnlineAnalysis: MOA (Massive On-line Analysis).
-----       1.0.3       No  metaCost: This metaclassifier makes its base classifier cost-sensitive using Pedro Domingo's method.
-----       1.0.1       No  multiBoostAB: Class for boosting a classifier using the MultiBoosting method.
-----       1.0.5       No  multiInstanceFilters: A collection of filters for manipulating multi-instance data.
-----       1.0.7       No  multiInstanceLearning: A collection of multi-instance learning classifiers.
-----       1.0.7       No  multiLayerPerceptrons: This package currently contains classes for training multilayer perceptrons with one hidden layer for classification and regression, and autoencoders.
-----       1.0.1       No  multilayerPerceptronCS: An extension of the standard MultilayerPerceptron classifier in Weka that adds context-sensitive Multiple Task Learning (csMTL)
-----       1.0.1       No  naiveBayesTree: Class for generating a decision tree with naive Bayes classifiers at the leaves.
-----       1.0.1       No  normalize: An instance filter that normalize instances considering only numeric attributes and ignoring class index
-----       1.0.4       No  oneClassClassifier: Performs one-class classification on a dataset.
-----       1.0.5       No  optics_dbScan: The OPTICS and DBSCAN clustering algorithms
-----       1.0.1       No  ordinalClassClassifier: Meta classifier that allows standard classification algorithms to be applied to ordinal class problems.
-----       1.0.1       No  ordinalLearningMethod: An implementation of the Ordinal Learning Method (OLM)
-----       1.0.1       No  ordinalStochasticDominance: An implementation of the Ordinal Stochastic Dominance Learner
-----       1.0.1       No  paceRegression: Class for building pace regression linear models and using them for prediction.
-----       1.0.1       No  partialLeastSquares: Partial least squares filter and classifier for performing PLS regression.
-----       1.0.4       No  predictiveApriori: Class implementing the predictive apriori algorithm for mining association rules.
-----       1.0.2       No  prefuseGraph: A visualization component for displaying graphs that uses the prefuse visualization toolkit.
-----       1.0.2       No  prefuseGraphViewer: A Knowledge Flow visualization component for displaying trees and graphs that uses the prefuse visualization toolkit.
-----       1.0.2       No  prefuseTree: A visualization component for displaying trees that uses the prefuse visualization toolkit.
-----       1.0.1       No  probabilisticSignificanceAE: Evaluates the worth of an attribute by computing the Probabilistic Significance as a two-way function
-----       1.0.1       No  raceSearch: Races the cross validation error of competing attribute subsets.
-----       1.0.1       No  racedIncrementalLogitBoost: Classifier for incremental learning of large datasets by way of racing logit-boosted committees.
-----       1.0.1       No  realAdaBoost: Class for boosting a 2-class classifier using the Real Adaboost method.
-----       1.0.1       No  regressionByDiscretization: A regression scheme that employs any classifier on a copy of the data that has the class attribute (equal-width) discretized.
-----       1.0.1       No  ridor: An implementation of a RIpple-DOwn Rule learner.
-----       1.0.2       No  rotationForest: Ensembles of decision trees trained on rotated subsamples of the training data.
-----       1.0.0       No  sasLoader: SAS sas7bdat file reader
-----       1.0.5       No  scatterPlot3D: A visualization component for displaying a 3D scatter plot of the data using Java 3D.
-----       1.0.1       No  scriptingClassifiers: Wrapper classifiers for Jython and Groovy scripting code.
-----       1.0.1       No  sequentialInformationalBottleneckClusterer: Cluster data using the sequential information bottleneck algorithm.
-----       1.0.4       No  simpleCART: Class implementing minimal cost-complexity pruning.
-----       1.0.1       No  simpleEducationalLearningSchemes: Simple learning schemes for educational purposes (Prism, Id3, IB1 and NaiveBayesSimple).
-----       1.0.1       No  stackingC: Implements StackingC (more efficient version of stacking)
-----       1.0.0       No  supervisedAttributeScaling: A simple filter to rescale attributes to reflect their discriminative power.
-----       1.0.1       No  tabuAndScatterSearch: Search methods contributed by Adrian Pino (ScatterSearchV1, TabuSearch)
-----       1.0.1       No  tertius: Finds rules according to confirmation measure (Tertius-type algorithm)
-----       1.0.3       No  thresholdSelector: A metaclassifier that selecting a mid-point threshold on the probability output by a Classifier.
-----       1.0.0       No  tigerjython: TigerJython
-----       1.0.0       No  timeSeriesFilters: Time Series Filters
-----       1.0.15      No  timeseriesForecasting: Time series forecasting environment.
-----       1.0.3       No  userClassifier: Interactively classify through visual means.
-----       1.0.1       No  votingFeatureIntervals: Classification by voting feature intervals.
-----       1.0.1       No  wavelet: A filter for wavelet transformation.
-----       1.0.4       No  wekaServer: Simple server for executing Weka tasks.
-----       1.0.1       No  winnow: Implements Winnow and Balanced Winnow algorithms by Littlestone

"Best performance" Debug output gives worst performance

Weka version: 3.7.13
multisearch version: 2015.10.15

Debug output(using experimenter):

Selected <Use an artificial time index>
Running experiment: Runs from: 1 to: 1
Datasets: D:\Hans\Dropbox\CNTT\151001DataMining\weka\피자헛\전체매장\00 00\data\allBranches.arff
Custom property iterator: on
Custom property path:
1  weka.experiment.RandomSplitResultProducer::splitEvaluator RegressionSplitEvaluator: weka.classifiers.rules.ZeroR (version 48055541465867954)
Custom property name:classifier
Custom property values:
 1 weka.classifiers.meta.FilteredClassifier FilteredClassifier: No model built yet.
ResultProducer: RandomSplitResultProducer: -P 63.0 -R -W weka.experiment.RegressionSplitEvaluator --: <null Instances>
ResultListener: weka.experiment.InstancesResultListener@7b2368d9

Writing experiment copy
Reading experiment copy
Made experiment copy
RandomSplitResultProducer: setting additional measures for split evaluator
RegressionSplitEvaluator: In set classifier
Shutting down thread pool...
Starting thread pool with 6 slots...

weka.classifiers.meta.MultiSearch
=================================
Options: -E MAE -search "weka.core.setupgenerator.MathParameter -property classifier.minNumInstances -min 2.0 -max 5.0 -step 1.0 -base 10.0 -expression pow(BASE,I)" -sample-size 100.0 -log-file "D:\\Program Files\\Weka-3-7" -initial-folds 2 -subsequent-folds 10 -num-slots 6 -S 1 -W weka.classifiers.trees.M5P -output-debug-info -- -M 200.0

Step 1:


=== Initial space - Start ===
Determining best values with 2-fold CV in space:
1-dimensional space:
 - 1. dimension: classifier.minNumInstances, min: 2.0, max: 5.0, step: 1.0

Performance (5): 10.410146327267936 (MAE): cached=false
Performance (4): 9.712894579717481 (MAE): cached=false
Performance (3): 8.758684173044188 (MAE): cached=false
Performance (2): 7.865438691244595 (MAE): cached=false

Best performance:
Performance (5): 10.410146327267936 (MAE)

Result of Step 1: 2

=== Initial space - End ===

Center is on border of space.

Final result:2
Classifier: weka.classifiers.trees.M5P -M 100.0
Shutting down thread pool...
Done...

Best performance should be Performance (2): 7.865438691244595 (MAE), not Performance (5): 10.410146327267936 (MAE). Not sure whether Multisearch picks worst or best parameters.

Maven dependency

Hi,

I'm having an issue with adding the dependency to pom.xml.
Where exactly do I need to add this?

I don't have any experience with Maven. Any help would be much appreciated.