iohprofiler / iohexperimenter

Experimentation procedure for Iterative Optimization Heuristics

IOHexperimenter

Experimenter for Iterative Optimization Heuristics (IOHs), built in* C++.

• Documentation: https://iohprofiler.github.io/IOHexperimenter.
• Publication: https://arxiv.org/abs/1810.05281.
• Wiki page: https://iohprofiler.github.io.

IOHexperimenter provides:

• A framework to ease the benchmarking of any iterative optimization heuristic.
• Pseudo-Boolean Optimization (PBO) problem set (25 pseudo-Boolean problems).
• Integration of the well-known Black-black Optimization Benchmarking (BBOB) problem set (24 continuous problems).
• W-model problem sets constructed on OneMax and LeadingOnes.
• Integration of the Tree Decomposition (TD) Mk Landscapes problems.
• Submodular optimization problems, as seen in the GECCO '22 workshop.
• Several benchmark suites from the CEC conference.
• Flexible interface for adding new suites and problems.
• Advanced logging module that takes care of registering the data in a seamless manner.
• Data format is compatible with IOHanalyzer.

Available Problem Suites:

• BBOB (Single Objective Noiseless) (COCO)
• SBOX-COST (COCO)
• StarDiscrepancy
• PBO
• Submodular Graph Problems
• CEC 2013 Special Session and Competition on Niching Methods for Multimodal Function Optimization
• CEC 2022 Special Session and Competition on Single Objective Bound Constrained Numerical Optimization

C++

The complete API documentation, can be found here, as well as a Getting-Started guide. In addition to the documentation, some example projects can be found in the example folder of this repository.

Python

The pip-version of IOHexperimenters python interface is available via pip. A tutorial with python in the form of a jupyter notebook can be found in the example folder of this repository. A Getting-Started guide and the full API documentation can be found here.

Contact

If you have any questions, comments or suggestions, please don't hesitate contacting us [email protected].

Our team

• Jacob de Nobel, Leiden Institute of Advanced Computer Science,
• Furong Ye, Leiden Institute of Advanced Computer Science,
• Diederick Vermetten, Leiden Institute of Advanced Computer Science,
• Hao Wang, Leiden Institute of Advanced Computer Science,
• Carola Doerr, CNRS and Sorbonne University,
• Thomas Bäck, Leiden Institute of Advanced Computer Science,

When using IOHprofiler and parts thereof, please kindly cite this work as

Jacob de Nobel, Furong Ye, Diederick Vermetten, Hao Wang, Carola Doerr and Thomas Bäck, IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics, arXiv e-prints:2111.04077, 2021.

@ARTICLE{IOHexperimenter,
  author = {Jacob de Nobel and
               Furong Ye and
               Diederick Vermetten and
               Hao Wang and
               Carola Doerr and
               Thomas B{\"{a}}ck},
  title = {{IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics}},
  journal = {arXiv e-prints:2111.04077},
  archivePrefix = "arXiv",
  eprint = {2111.04077},
  year = 2021,
  month = Nov,
  keywords = {Computer Science - Neural and Evolutionary Computing},
  url = {https://arxiv.org/abs/2111.04077}
}

iohexperimenter's Issues

failed to clone the fmt file

Hello all
I'm trying to install the IOHexperimenter onto my fedora linux machine and its having issues with cloning in the fmt

Cloning into '/path/IOHexperimenter/external/fmt'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:fmtlib/fmt' into submodule path '/pathIOHexperimenter/external/fmt' failed
Failed to clone 'external/fmt'. Retry scheduled

it then tries again as scheduled and still fails

I just wanted to know if anyone else had similar issues and if they knew a fix

Limited precision in data-files

When running BBOB-problems, the data-files we store should not be limited by a low numerical precision (e.g. 1e-7 should not be written as 0.000000, as this causes an information loss).
This could be solved by always (or at least when dealing with this kind of precision to target values) converting our objective value to scientific (exponential) notation.

no member named 'isnan' in namespace 'std'; did you mean simply 'isnan'?

I try to make the project but I face the following error, I'm using Mac,

./../Problems/BBOB/f_rastrigin.hpp:69:9: error: no member named 'isinf' in
      namespace 'std'; did you mean simply 'isinf'?
    if (std::isinf(sum2)) {
        ^~~~~~~~~~
        isinf
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/math.h:497:1: note: 
      'isinf' declared here
isinf(long double __lcpp_x) _NOEXCEPT { return __libcpp_isinf(__lcpp_x); }
^
In file included from IOHprofiler_BBOB_suite.hpp:25:
./../Problems/BBOB/f_rastrigin_rotated.hpp:102:9: error: no member named 'isinf'
      in namespace 'std'; did you mean simply 'isinf'?
    if (std::isinf(sum2)) {
        ^~~~~~~~~~
        isinf
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/math.h:497:1: note: 
      'isinf' declared here
isinf(long double __lcpp_x) _NOEXCEPT { return __libcpp_isinf(__lcpp_x); }
^
In file included from IOHprofiler_BBOB_suite.hpp:28:
./../Problems/BBOB/f_schaffers10.hpp:99:11: error: no member named 'isinf' in
      namespace 'std'; did you mean simply 'isinf'?
      if (std::isinf(tmp) && std::isnan(sin(50.0 * pow(tmp, 0.1))))  /* ...
          ^~~~~~~~~~
          isinf
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/math.h:497:1: note: 
      'isinf' declared here
isinf(long double __lcpp_x) _NOEXCEPT { return __libcpp_isinf(__lcpp_x); }
^
In file included from IOHprofiler_BBOB_suite.hpp:28:
./../Problems/BBOB/f_schaffers10.hpp:99:30: error: no member named 'isnan' in
      namespace 'std'; did you mean simply 'isnan'?
      if (std::isinf(tmp) && std::isnan(sin(50.0 * pow(tmp, 0.1))))  /* ...
                             ^~~~~~~~~~
                             isnan
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/math.h:541:1: note: 
      'isnan' declared here
isnan(long double __lcpp_x) _NOEXCEPT { return __libcpp_isnan(__lcpp_x); }
^
In file included from IOHprofiler_BBOB_suite.hpp:29:
./../Problems/BBOB/f_schaffers1000.hpp:99:11: error: no member named 'isinf' in
      namespace 'std'; did you mean simply 'isinf'?
      if (std::isinf(tmp) && std::isnan(sin(50.0 * pow(tmp, 0.1))))  /* ...
          ^~~~~~~~~~
          isinf
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/math.h:497:1: note: 
      'isinf' declared here
isinf(long double __lcpp_x) _NOEXCEPT { return __libcpp_isinf(__lcpp_x); }
^
In file included from IOHprofiler_BBOB_suite.hpp:29:
./../Problems/BBOB/f_schaffers1000.hpp:99:30: error: no member named 'isnan' in
      namespace 'std'; did you mean simply 'isnan'?
      if (std::isinf(tmp) && std::isnan(sin(50.0 * pow(tmp, 0.1))))  /* ...
                             ^~~~~~~~~~
                             isnan
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/math.h:541:1: note: 
      'isnan' declared here
isnan(long double __lcpp_x) _NOEXCEPT { return __libcpp_isnan(__lcpp_x); }

pip install fails for Python 3.9 on Linux

A wheel file for Linux for Python 3.9 (cp39) is missing from PyPi.

It also rejects building all the source tarballs because the version in metadata differs from the PyPi specified one. With an error like:

Requested ioh (...)ioh-0.3.2.1.tar.gz (...) has different version in metadata: '0.2.9.9'

This causes pip to eventually fall back to the source file for 0.0.0.1 which then fails to build.

A workaround for Linux users is to use Python 3.8, or compile their own wheel. The latter runs into similar problems with version needing some manual adjustment, but the former works just fine.

Double-Check Implementation of Ising Model

If I understand correctly, then the Ising model is defined as follows:

We have a graph where the nodes have the names 1 to N.
These nodes are connected by the set of edges E.
The "value" of a node i be its value in the candidate solution x, i.e., x[i] .
The objective value is the number of edges where both involved nodes have the same value.
This gives us a global optimum at the all-zeros string and the all-ones string.

Roughly speaking, we should have pseudo code like this:

f <- 0
for all Edges e do
  i <- start of edge e
  j <- end of edge e
  if x[i] == x[j] then f <- f + 1
end

The maximum objective value should be N and the minimum objective value should be 0.

However, when I look at your implementations of the Ising model, I think -- and I might be wrong, but well, I think -- that you count all edges twice, at least in the 1D version(?)
See, e.g.,

IOHexperimenter/src/Problems/f_ising_1D.hpp

Line 43 in 7b97260

double internal_evaluate(const std::vector<int> &x) {

  double internal_evaluate(const std::vector<int> &x) {
    int result= 0, n = x.size();

    for (int i = 0; i < n; ++i) {
        int first_neig=x[modulo_ising_1D((i+1), n)];
        int second_neig=x[modulo_ising_1D((i -1), n)];

        result += (x[i] *first_neig) + ((1- x[i])*(1- first_neig));
        result += (x[i] *second_neig) + ((1- x[i])*(1- second_neig));
    }
    return (double)result;
  };

For example, if n = 3, then on a 1-D Torus I should have three edges, namely 0-1, 1-2, and 2-0.
The optimal objective value should then probably be 3, I think.
However, if I understand correctly, then the loop will go from i=0 to i=2 and would probably do:

// i = 0
result += (x[0] *x[1]) + ((1- x[0])*(1- x[1]));
result += (x[0] *x[2]) + ((1- x[0])*(1- x[2]));
// i = 1
result += (x[1] *x[2]) + ((1- x[1])*(1- x[2]));
result += (x[1] *x[0]) + ((1- x[1])*(1- x[0]));
// i =2
result += (x[2] *x[0]) + ((1- x[2])*(1- x[0]));
result += (x[2] *x[1]) + ((1- x[2])*(1- x[1]));

This seems to include each index pair twice, and if I have the all-ones string, I should get 6 as result instead of 3.

Again: Maybe I misunderstood something elementary here.
Also: From the optimization perspective, this should not really matter.
But it might matter if someone else implements the Ising model, does own experiments, and then wants to compare the results with the output of the IOHexperimenter.
If we want to compare the expected running times to reach certain objective values, then this may lead to a confusion.

Side note 1: I am not sure whether the compiler allows you to do that, but maybe doing (x[i]==x[j]) & 1 could be a dodgy but fast way to compute x[i]*x[j]+(1-x[i])*(1-x[j]).

Side note 2: IOHprofiler/IOHprofiler.github.io#2

test_wrap_problem sometimes segfaults

this happens in the CI sometimes with different python versions and different operating systems. But often, when you run it again the error disapears.

test_analyzer (tests.python.test_logger.TestLogger) ... ok
test_eaf (tests.python.test_logger.TestLogger) ... ok
test_flatfile (tests.python.test_logger.TestLogger) ... ok
test_pickle (tests.python.test_logger.TestLogger) ... ok
test_bbob_problems_first_instance (tests.python.test_problem.TestProblem) ... ok
test_evaluation_bbob_problems (tests.python.test_problem.TestProblem) ... ok
test_evaluation_pbo_problems (tests.python.test_problem.TestProblem) ... ok
test_experimenter (tests.python.test_problem.TestProblem) ... ok
test_file_comparisons (tests.python.test_problem.TestProblem) ... ok
test_get_problem (tests.python.test_problem.TestProblem) ... ok
test_pbo_problems_first_instance (tests.python.test_problem.TestProblem) ... ok
test_wmodel (tests.python.test_problem.TestProblem) ... Segfault

This could be to something in the test_wmodel test, but also it could be an error in the general cleanup methods, since this is the last test to run for the python part. We should check the registered atexit methods.

IOH package installation

Description: Errors on installing IOH python package.

Versions: 0.3.2.7.2

Steps to Reproduce: running 'python -m pip install IOH'

Actual Behavior: The package can not be installed successfully.

We should have min/max target estimates of problems

On a separate accessors than optimum.

[3d party] namespaces and class names

While implementing a third party solver that use IOH as a problem provider (along with using logging), I faced the following problems (based on the developing branch, as of commit e46996f).

All the classes are prefixed with IOHprofiler. Many methods are also prefixed, giving some cumbersome calls, like IOH_problem* bench = new BBOB_suite; auto bench_problems = bench->IOHprofiler_suite_get_problems();.

It would be more intuitive for a new user (at least to me) to use namespaces and avoid member names prefixed with the class name: ioh::Suite* bench = new ioh::suite::BBOB; auto bench_problems = bench->get_problems();. As a plus, this gives the opportunity to using namespace ioh; if concision is wanted.

Cannot use None with property in python.

Because of this, we directly cast to double, without checking for None values:

IOHexperimenter/ioh/src/logger.cpp

Line 28 in e28136a

return std::make_optional<double>(PyFloat_AsDouble(container_.attr(attribute_.c_str()).ptr()));

Add macro for format header only.

Add #define FMT_HEADER_ONLY

Before including fmt headers. This will ensure that users can include the repository more easily.

Logger cannot merge runs in output data when called out of context

I'm interfacing my solver with IOHexperimenter at the problem level, because I wish to use my own experiment manager. Indeed, I wish to use my own batch system to place (down to) single runs as independent jobs on an arbitrary set of machines. However, this is not feasible with the current architecture.

The core of the problem is that the Logger interface expect to be initialized once, and then to be called on different target_problem, for several runs. However, if one wish to handle jobs at the runs level, the Logger will be initialized for each run. In that setting, the Logger avoid adding the results in the previously written data folder, and create an additional folder, leading to two different folders holding two separated runs.

Unfortunately, this data structure cannot be loaded as a single experiment by IOHanalyzer.

To test the problem, merge PR #38 and run t-bbob-csvlogger twice.

The solutions I can imagine are to:

• add a state manager to Logger that would allow to load previous data structures from saved files,
• allow IOHanalyzer to parse several folders as a single experiment,
• handle this specific case with an additional data merger.

Problem with bounds

In IOHprofiler_problem.hpp, the functions IOHprofiler_set_number_of_variables set the entire upper/lowerbounds to their first values, cast to an integer. This does not allow for non-box constraints or real-valued bounds. For example, the bound [1.5, 2, 3.5] is changed to [1, 1, 1]. These lines are responsible (in both overloads):

if (this->lowerbound.size() != 0) {
  this->IOHprofiler_set_lowerbound(this->lowerbound[0]);
}
if (this->upperbound.size() != 0) {
  this->IOHprofiler_set_upperbound(this->upperbound[0]);
}

Removing these lines solved the problem for me.

Create a chance to advance the `timestep`. Randomize ordering of weights in `DynamicBinVal`, when `timestep` changes.

ECDF logger log range is wrong when maximizing

The (default) log-log range computation for the discretization of the attainment matrices assume minimization. It should inverse the log axis in case of maximization.

Error, warning and log messages handling

As of today, there's no modern handling of error or warning messages.
The entry point is in the common file.

Problems of this approach are:

• The IOH_error function does exit the program, however, raising an exception may be easier to handle for higher layers, especially for bindings toward other languages.
• The IOH_warning and IOH_log output to cout, which breaks common assumptions.
• Only one IOH_log overload allows for a (hard-coded) redirection on another stream, which may pollute any embedding in an interactive tool.
• There is no log level management, which makes debugging difficult (incidentally, this may explain the lack of use of the log function?).
• The functions take a string, making it mandatory to forge a message prior to their call if one want to output a variable's value.

I suggest the following refactoring:

• Use the clutchlog library (which I developed with IOH in mind).
• Allow to set up a default redirection to a file stream (for bindings).
• Raise an exception in IOH_error.
• Replace IOH_warning and IOH_log by a single IOH_log macro, taking a log level and a stream, which may be used as: IOH_log(warning, "Maybe you don't want to have " << that_many << " objectives");.
• Use the logging feature a lot more in the code to enable easier debugging.

Split tests executables

The current setup is to have a single executable which holds all the tests.

Problem

This is not scalable. When one develop a new feature, one would expect to be able to do rapid build/test iterations, for instance by building only the related test(s). Having all in one place just makes the builds longer, reducing efficiency.
Additionally, when tests run are going to become longer, a similar problem will occur with runtimes. While being able to run only a targeted set of tests would also solve the problem.

Suggested solution

Use a single executable target by test and let a test manager (like CTest, as the project is using CMake anyway) allow for various test execution options (tests filtering patterns, parallel runs, code analysis integration, dashboard submissions, etc.).

Add `DynamicBinVal` problem to be among the problems in IOHexperimenter.

Can't instantiate custom suite from configuration.ini

Hi, I have implemented a custom IOHprofiler_suite following the instructions at the wiki. Now I am trying to use it by passing the suite name to suite_name in configuration.ini. I first register the suite using the statement:

static registerInFactory<IOHprofiler_suite<double>,Random_suite> regSuite("RANDOM");

"RANDOM" is the name of the suite. Now, I initialize the IOHprofiler_experimenter using:
IOHprofiler_experimenter<double> experimenter(configFile,algorithm);

This throws the error: IOH_ERROR_INFO : Input value exceeds upperbound.. I think this error is thrown because the suite is not registered correctly, because neither of the constructors of the suite is ever called. Also, instantiating the suite 'manually' like this does work:
std::shared_ptr<IOHprofiler_suite<double>> suite = genericGenerator<IOHprofiler_suite<double>>::instance().create("RANDOM");

Am I doing something wrong, or is it not possible to instantiate custom suites via the configuration file? I have the latest version of IOHexperimenter installed. Thanks in advance!

Missing optimum(s) for WModel and NKL problems

WModel and NKLandscapes problems are missing known optima computation.

I remind that it was possible to deduce them by simple computations from the configuration and add the results in objective_ at construction, but this is currently missing.

This information is however crucial for using correctly the Empirical Attainment loggers, without it, it is probably a far less useful feature.

License missing

A license file is missing.
Based on IOHanalyzer's README, I guess it may be BSD, but a LICENSE file and headers would make it clear.

Add a Python binding for `DynamicBinVal`.

Problem with pip install ioh

Dear sir/ madam,

I got an error when I was trying to download the latest version of ioh. This error disappeared when I downloaded the second latest version. Do you know how to solve this?

Kind Regards,
Rens

Star Discrepancy optimum value

when calling problem.optimum on any star discrepancy the output looks like this:

Which is obviously not the right answer. Tested on Star Discrepancy Integer and Star Discrepancy Real and encountered the problem on both.

Assign optimum in Python Classes

In the Python interface, we cannot assign the optimum for new problems that are defined as a class.

An argument should be added to the super constructor.

Parallellism-related file writing issues

When using the Experimenter-class in the python version of ioh, there is an option for parallellism.
When enabled, this can lead to broken info and data-files, which does not happen when executing the same code sequentially.
The issues seen so far include (See pyadede_mirror.zip for example of resulting files, generated with merge_output=True):

• Missing runs (~40 out of the required 50 runs were present for each function).
• Empty run-lines in info-files (after the .dat filename, there is a ', ,' where run-information should be
• Data from other runs inserted into the data-file (judging by the corresponding positions, this is data from another run/function)

Code used to generate these issues:
exp = ioh.Experiment(algorithm = Optimizer(), #Set the optimization algorithm, can be arbitrary fids = range(1,25), iids = [1,2,3,4,5], dims = [5,10], reps = 5, problem_type = 'BBOB', #Problem definitions njobs = 64, #Enable paralellization logged = True, folder_name = 'temp', algorithm_name = '{temp}', store_positions = True, #Logging specifications merge_output = False, zip_output = False, remove_data = False #Any setting works, set merge to false to be able to more easily find where the errors occur ) exp()

observer call interface

As of today, the only logger class (IOHprofiler_csv_logger) is documented to be actually called with logger.write_line(…).

However, its base class (IOHprofiler_observer) do not expose such a call interface.

Consequently, it is not actually possible to declare a generic binding to any (future) class of observer.

I suggest to add a generic call interface, like virtual void IOHprofiler_observer::log(const std::vector<double> &logger_info) = 0; and replace write_line in examples.

BBOB problem dimension limited to 54

(Some of?) the BBOB suite problems are limited to 54 dimensions, due to the implementation of bbob2009_gauss in ./src/Problems/BBOB/bbob_common_used_functions/suite_bbob_legacy_code.hpp:97.

Either this is actually wanted and a check on dimension should occur earlier in the call stack, either the implementation should be scalable.

Replace in folder names

This string replace in folder names also replaces IOHprofiler higher up the path, it should only do it for the lowest path.

IOHexperimenter/ioh/__init__.py

Line 423 in c0d44c5

source_dat_folder = os.path.splitext(source)[0].replace(

Loggers design issues

The Analyzer logger takes triggers as a first parameter, which effectively forces a naive user to pass at least the on_improvement trigger for the logger to work. I would rather see the triggers defaulted near the end of the constructor parameters, so that a naive user would not have to bother with this feature (which is more for advanced users).

The Analyzer also makes a difference between Properties and Attributes, but I don't understand why. analyzer::structures::Attribute stores a key-value pair, I guess to log the algorithm's parameters. But this is also the objective of Properties, why have a specific class for Analyzer? The aim of properties is also to allow to log any algorithm parameter in any logger.

The properties were also designed so as to be a generic view about anything that would be logged. In that sense, they are also a way to control if the x are stored (or not). But loggers have a store_position flag, which necessitates additional code to check if one wants to log log_info.current.x. It would seem more logical to let a property handle this. That way, one does not need additional code, but only to pass (or not) a Property handling the current x.
However, given that a constructor with a boolean is surely easier to grasp by a newcomer, we should keep a separated —simplified— constructor with this flag, and add the related property if it is true.

Additionally, hiding the default necessary properties is a good idea, and I would suggest going further so as to do the same with triggers: have the on_improvement as hidden default, and allow for additional triggers in the full constructor.

The constructor that I would find easier to grasp for a newcomer would put the most sensible parameters first, something like:

//! Simple constructor.
Analyzer(
               const std::string &folder_name = "ioh_data",
               const std::string &algorithm_name = "algorithm_name",
               const std::string &algorithm_info = "algorithm_info",
               const fs::path &root = fs::current_path(),
               const bool store_positions = false,
               const Properties &additional_properties = {}
    ) {}
//! Constructor with full control.
Analyzer(
               const std::string &folder_name = "ioh_data",
               const std::string &algorithm_name = "algorithm_name",
               const std::string &algorithm_info = "algorithm_info",
               const fs::path &root = fs::current_path(),
               const Properties &additional_properties = {},
               const Triggers &additional_triggers = {}
    ) {}

Explicit interface contracts

It's somehow difficult to infer what is the contract that additional features should implement in the generic interfaces. If someone want to add a new problem to IOH, this renders the exercise more complicated.

For instance, the IOHprofiler_problem class has virtuals functions, but the internal_call is not pure virtual, and raise an error if called. An elegant implementation followings C++ coding style would be to remove the useless implementation and makes it = 0.

Additionally, it is not clear if some part of the interface is mandatory or not. For instance, the prepare_problem function is virtual, but no documentation indicates what is expected with it.

An improvement would be to use abstract classes, so as to makes clear and enforce the intended interfaces of the classes in the Template directory.
I suggest that the expected interface functions are moved at the beginning of the headers, with documentation explaining what is expected.

Default directories of csv logger cannot be created

If one instantiates a csv_logger with the empty constructor, the program crashes because it cannot create a hierarchy of directories.

Creating the top-level directory by hand solves the problem, but violates user expectations.

Using C++17's filesystem module instead of the adhoc functions in IOHprofiler_csv_logger.cpp would solve such problems.

Too many warnings

There are far too many warnings when compiling IOHexperimenter with the usual -Wall -Wextra -Wpedantic flags. This renders compiler's output difficult to read, while most of them are -Wsign-compare, -Wunused-parameter or -Wunused-function, and thus should be easy to fix.

Incorrect epistasis layer for W-model problem.

https://github.com/IOHprofiler/IOHexperimenter/blob/master/include/ioh/problem/wmodel/wmodel_problem.hpp#L86
the input and result binary vectors are the same, so during the epistasis evaulation, the intermediate results affect the final result.
Due to this, the target bit string and fitness values of the W-model problem with epistasis values higher than 2 are incorrect.

Simple Example (see Fig.2 in Weise and Wu, 2018):
epi = 4
in = [1, 0, 0, 0]
current implementation works as follows:

1. in = 1000, set 1 to 4th bit (1 xor 0 xor 0 xor 0 = 1)
2. in = 1001, set 1 to 3th bit (1 xor 0 xor 0 = 1)
3. in = 1011, set 0 to 2nd bit (1 xor 1 xor 0 = 0)
4. in = 1011, set 1 to 1st bit (1 xor 1 xor 1 = 1)

actual result = [1, 0, 1, 1] with fitness value of 3.

expected implementation should have input and result vectors different:
in = [1, 0, 0, 0]
res = [0, 0, 0, 0]

1. res = 0000, set 1 to 4th bit (1 xor 0 xor 0 xor 0 = 1)
2. res = 0001, set 1 to 3th bit (1 xor 0 xor 0 = 1)
3. res = 0011, set 1 to 2nd bit (1 xor 0 xor 0 = 1)
4. res = 0111, set 1 to 1st bit (1 xor 0 xor 0 = 1)

expected res = [1, 1, 1, 1]

Feature: objective function as a service

IOHexperimenter should have a high-level binary implementing an "objective function as a service".
This would take the form of an executable that takes the problem configuration (suite, problem id, instance) as parameters and then watch I/O named pipes (cf. mkfifo in POSIX systems) as objective function calls.

For reference, I've implemented a generic proof of concept for Linux in nojhan/named-pipes-services. Thanks to the blocking nature of named pipes, the code is utterly simple and basically consists of a loop: reading an input file, calling the function, then writing to a file.
Such a service can then be very easily exposed on a network with the help of classical tools like socat.

Necessary steps:

• I/O data structures (plain CSV? JSON?),
• command-line arguments management library (boost? single-header? sub-module?),
• implement,
• implement tests,
• write tutorial.

Assertion in wrap problem

In [1]: import ioh

In [2]: def f(x): return 0.0

In [3]: ?ioh.problem.wrap_real_problem
Docstring: wrap_real_problem(f: function, name: str, n_variables: int = 5, optimization_type: ioh.iohcpp.OptimizationType = <OptimizationType.Minimization: 0>, constraint: ioh.iohcpp.RealConstraint = <Constraint lb: [{-1.7976931348623157e+308, -1.7976931348623157e+308, -1.7976931348623157e+308, -1.7976931348623157e+308, -1.7976931348623157e+308}] ub: [{1.7976931348623157e+308, 1.7976931348623157e+308, 1.7976931348623157e+308, 1.7976931348623157e+308, 1.7976931348623157e+308}]>) -> ioh.iohcpp.problem.RealWrappedProblem
Type:      builtin_function_or_method

In [4]: ioh.problem.wrap_real_problem(f, "f1")
Out[4]: <RealProblem 25. f1 (iid=0 dim=5)>

In [5]: ioh.problem.wrap_real_problem(f, "f1")
python: /tmp/pip-req-build-0jvb2lgv/include/ioh/common/factory.hpp:89: void ioh::common::Factory<AbstractType, Args>::include(const string&, int, ioh::common::Factory<AbstractType, Args>::Creator) [with AbstractType = ioh::problem::Problem<double>; Args = {int, int}; std::string = std::basic_string<char>; ioh::common::Factory<AbstractType, Args>::Creator = std::function<std::shared_ptr<ioh::problem::Problem<double> >(int&&, int&&)>]: Assertion `!already_defined' failed.
Aborted

[feature request] Easier logging of separate functions with a single logger

It would be useful to have the ability to use one logger to run on multiple functions sequentially. While this works for making the data files, currently only the first function ran this way gets an info file.

[3d party] headers inclusion scheme

While implementing a third party solver that use IOH as a problem provider (along with using logging), I faced the following problems (based on the developing branch, as of commit e46996f).

Problem: the header inclusion scheme makes it very painful to include the necessary files. Every include in the source expects a relative flat hierarchy, while the sources are deeply nested.

In order to be able to easily include from third party app, we would need to #include <IOH/Template/problem.hpp>. Having a sub-directory avoid file name collision.

To achieve that, I see two easy ways:

• either use an "installation" scheme, and add an installation target that copy the headers hierarchy in some location, containing an "IOH" directory (think /usr/local/include/IOH/…).
• either use a "framework" scheme, and rename the current "src" directory to "IOH" (or IOHexperimenter, of course). That way, third party apps just have to include_dir the root dir and #include <IOH/…> in their source.

Of course, one can do both.

Separate CEC 2022 transformation code from CEC 2022 function code into two different files: `functions/real.hpp` and `transformation.hpp`.

Test and approve the logic and correctness of CEC 2022 functions.

Test and approve the logic and correctness of CEC 2022 functions.

• From Python
• From C++

Add doxygen documentation to CEC 2022 functions.

master won't build with clang 9.0

A fresh master clone at 33342bd does not build with clang 9.0.1-12 on x86_64-pc-linux-gnu (Ubuntu 20.04 LTS) because of pybind11:

$ cmake 
$ make

[ 80%] Building CXX object ioh/CMakeFiles/iohcpp.dir/src/logger.cpp.o
cd /home/nojhan/code/IOHexperimenter/debug/ioh && /usr/bin/c++ -DFMT_HEADER_ONLY=1 -DPYTHON_DEFINED -DVERSION_INFO="" -Diohcpp_EXPORTS -I/home/nojhan/code/IOHexperimenter/include -I/home/nojhan/code/IOHexperimenter/external/fmt/include -isystem /usr/include/python3.8 -isystem /home/nojhan/code/IOHexperimenter/external/pybind11/include -g -fPIC -fvisibility=hidden -Wno-register -Wno-deprecated-register -std=gnu++17 -MD -MT ioh/CMakeFiles/iohcpp.dir/src/logger.cpp.o -MF CMakeFiles/iohcpp.dir/src/logger.cpp.o.d -o CMakeFiles/iohcpp.dir/src/logger.cpp.o -c /home/nojhan/code/IOHexperimenter/ioh/src/logger.cpp
In file included from /home/nojhan/code/IOHexperimenter/ioh/src/logger.cpp:4:
In file included from /home/nojhan/code/IOHexperimenter/include/ioh.hpp:3:
In file included from /home/nojhan/code/IOHexperimenter/include/ioh/common.hpp:4:
In file included from /home/nojhan/code/IOHexperimenter/include/ioh/common/file.hpp:3:
/home/nojhan/code/IOHexperimenter/include/ioh/common/utils.hpp:363:44: warning: unknown warning group '-Wformat-truncation', ignored [-Wunknown-warning-option]
            #pragma GCC diagnostic ignored "-Wformat-truncation"
                                           ^
In file included from /home/nojhan/code/IOHexperimenter/ioh/src/logger.cpp:1:
In file included from /home/nojhan/code/IOHexperimenter/external/pybind11/include/pybind11/pybind11.h:48:
/home/nojhan/code/IOHexperimenter/external/pybind11/include/pybind11/detail/init.h:63:68: error: no matching constructor for initialization of 'ioh::logger::Default'
inline Class *construct_or_initialize(Args &&...args) { return new Class{std::forward<Args>(args)...}; }
                                                                   ^    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/nojhan/code/IOHexperimenter/external/pybind11/include/pybind11/detail/init.h:175:31: note: in instantiation of function template specialization 'pybind11::detail::initimpl::construct_or_initialize<ioh::logger::Default, std::filesystem::__cxx11::path, std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char>, bool, bool, int, int, bool, std::vector<int, std::allocator<int> >, ioh::common::OptimizationType, int, int, 0>' requested here
            v_h.value_ptr() = construct_or_initialize<Cpp<Class>>(std::forward<Args>(args)...);
                              ^
/home/nojhan/code/IOHexperimenter/external/pybind11/include/pybind11/pybind11.h:1346:14: note: in instantiation of function template specialization 'pybind11::detail::initimpl::constructor<std::filesystem::__cxx11::path, std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char>, bool, bool, int, int, bool, std::vector<int, std::allocator<int> >, ioh::common::OptimizationType, int, int>::execute<pybind11::class_<ioh::logger::Default, ioh::logger::Base, std::shared_ptr<ioh::logger::Default> >, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>' requested here
        init.execute(*this, extra...);
             ^
/home/nojhan/code/IOHexperimenter/ioh/src/logger.cpp:49:10: note: in instantiation of function template specialization 'pybind11::class_<ioh::logger::Default, ioh::logger::Base, std::shared_ptr<ioh::logger::Default> >::def<std::filesystem::__cxx11::path, std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char>, bool, bool, int, int, bool, std::vector<int, std::allocator<int> >, ioh::common::OptimizationType, int, int, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>' requested here
        .def(
         ^
/home/nojhan/code/IOHexperimenter/include/ioh/logger/default.hpp:230:18: note: candidate constructor not viable: no known conversion from 'bool' to 'const common::OptimizationType' for 5th argument
        explicit Default(fs::path output_directory = fs::current_path(), const std::string &folder_name = "ioh_data",
                 ^
/home/nojhan/code/IOHexperimenter/include/ioh/logger/default.hpp:248:18: note: candidate constructor not viable: requires single argument 'conf', but 13 arguments were provided
        explicit Default(const experiment::Configuration &conf) :
                 ^
/home/nojhan/code/IOHexperimenter/include/ioh/logger/default.hpp:14:11: note: candidate constructor (the implicit copy constructor) not viable: requires 1 argument, but 13 were provided
    class Default final : public Base, Observer
          ^
1 warning and 1 error generated.

With the following build options:

$ make -L ..

Re-run cmake no build system arguments
-- Version: 7.1.3
-- Build type: Debug
-- CXX_STANDARD: 17
-- Required features: cxx_variadic_templates
CMake Deprecation Warning at external/googletest/CMakeLists.txt:4 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Deprecation Warning at external/googletest/googlemock/CMakeLists.txt:45 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Deprecation Warning at external/googletest/googletest/CMakeLists.txt:56 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- pybind11 v2.6.2 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/nojhan/code/IOHexperimenter/debug
-- Cache values
BUILD_DOCS:BOOL=OFF
BUILD_EXAMPLE:BOOL=ON
BUILD_GMOCK:BOOL=ON
BUILD_PYTHON_PACKAGE:BOOL=ON
BUILD_TESTS:BOOL=ON
CMAKE_BUILD_TYPE:STRING=Debug
CMAKE_INSTALL_PREFIX:PATH=/usr/local
ENABLE_PYTHON_PROBLEMS:BOOL=ON
FMT_CUDA_TEST:BOOL=OFF
FMT_DEBUG_POSTFIX:STRING=d
FMT_DOC:BOOL=OFF
FMT_FUZZ:BOOL=OFF
FMT_INC_DIR:STRING=include
FMT_INSTALL:BOOL=OFF
FMT_OS:BOOL=ON
FMT_PEDANTIC:BOOL=OFF
FMT_TEST:BOOL=OFF
FMT_WERROR:BOOL=OFF
INSTALL_GTEST:BOOL=ON
PYBIND11_FINDPYTHON:BOOL=OFF
PYBIND11_INSTALL:BOOL=OFF
PYBIND11_NOPYTHON:BOOL=OFF
PYBIND11_PYTHON_VERSION:STRING=
PYBIND11_TEST:BOOL=OFF

The BUILD_PYTHON_PACKAGE obviously triggers the problem.

Faster objective function value comparison

As of now, objective function value comparison goes through the common::OptimizationType enum, which is passed to functions like compare_objectives, where there is a test, which branch out on operator< or operator>.

The problem is that such comparisons are heavily used in the algorithm that computes the empirical attainment function.
Hence, having a test at each call is heavy on CPU.

It should be possible to optimize out the test by letting the OptimizationType holds a comparison operator, which would be directly used. Or use the the enum as a template parameter to specialize the comparison functions and let the compiler plug the good one.

Both options may restrict loggers to homogeneous problem sets.
In the first case, the virtual indirection may be not worth the test, so maybe the second option would be better.

STAR discrepancy metadata information

In python module, please add the following information for meta_data.optimization_type, maximisation or minimisation.

Problem iid 0 assigned when adding new problems

When I add two or more customised functions by

experiment.add_custom_problem(my_opt_1, "my_opt_1_function")

experiment.add_custom_problem(my_opt_2, "my_opt_2_function")

The experimenter exactly runs both two function. But in the data file, there is a result of only one function. I gauss that since “fid” are same in both function (f0), one result has been overridden.

Stubs are not updated correctly

Some type defintions are missing, and some stubs have incomplete types.

General review after the "restructure" branch refactoring

Those are general comments, after having done a high-level review of the whole code:

• There is far too few documentation and comments within certain module of code (experimenter, random, utils, etc.).
• I do not quite understand the random class. It's also full of magic numbers and some of them may be replaced by some std::numeric_limits.
• I'm still puzzled by the concrete meaning of the parameters of the logger::csv constructor, some doc here would help.
• There is no clear style as whether to go "headers only" or to separate code that should be built up in a library. A lot of classes have embedded compilable code. I'm not sure which one is best (maybe headers-only, in IOH case) but it would help to stick to one style.
• A CONTRIBUTING.md file indicating the code style (tabs, namespaces, etc.) and expected PR workflow would be a plus.

Pypi package error?

Hi,
IOHProfiler package seems to be broken:
ERROR: Requested IOHexperimenter>=0.2.8 from https://files.pythonhosted.org/packages/83/9d/97995997158dd9bf80f8731fc4d3eaf4ebbf1e787505a48745a32e8744f0/IOHexperimenter-0.2.8r.tar.gz#sha256=837ba5e8844de28344bfde0810a19e6fa9e91ba9eddc11f19ac5be796f4db0a2 (from nevergrad==0.4.2.post2) has different version in metadata: '0.2.8'

I recently got this error both on Windows and Ubuntu. Eg:
https://app.circleci.com/pipelines/github/jrapin/nevergrad/22/workflows/a953c962-8a90-4823-bdb3-f787d9399dd7/jobs/98

Feature: design of experiment pipelining

IOHexperimenter should have simple workflows for answering the following question: "is my solver better than state of the art?".

This requires an automated experimental pipeline: run an algorithm configuration to tune the solver's parameter, then estimate its performances on a benchmark (with optional cross-validation).
To be as simple as possible, this experimental pipeline should be able to run seamlessly on an HPC cluster, without much intervention from the user.

A good way to do that would be to use modern workflow managers, like SnakeMake or Nextflow, or implement the design of experiment through openMole.

PointerReference nullptr issue on gcc

The test: store_properties currently fails.

It fails on the assertion that the (first_eval) value for:

double* p_transient_att = nullptr;
watch::PointerReference attpr("Att_PtrRef", p_transient_att);

is a std::nullopt, if left unchanged, since the pointer it refers to is set to a nullptr. When using other compilers (MSVC, Clang), this does not fail, and a std::nullopt is produced correctly.

The PointerReference gets constructed properly, and nullptr gets assigned to _variable. However, when calling the call operator, the value for this pointer member _variable, is changed to some other random memory adress, which cause the logic in the method to produce a random double value for that adress, as we just cast the result.

Since the field _variable is a protected member, I do not understand how its value can change outside the scope of the class body or its parents, but this still clearly happens.

Another weird thing about the bug is, it that there is a way to supress it; By calling the property once with a default logger::Info, the value _variable remains to be a nullptr:

logger::Info info;
attpr(info);

The above is currently added to the test store_properties in order to ensure we pass the CI.

(tagging @nojhan, @jdreo)