mlopez-ibanez / eaf

R package for plotting the Empirical Attainment Function

eaf: Empirical Attainment Function (EAF) Tools

[ Homepage ] [ GitHub ]

Maintainer: Manuel López-Ibáñez

Contributors: Manuel López-Ibáñez, Marco Chiarandini, Carlos M. Fonseca, Luís Paquete, Thomas Stützle, and Mickaël Binois.

---

Introduction

The empirical attainment function (EAF) describes the probabilistic distribution of the outcomes obtained by a stochastic algorithm in the objective space. This R package implements plots of summary attainment surfaces and differences between the first-order EAFs. These plots may be used for exploring the performance of stochastic local search algorithms for biobjective optimization problems and help in identifying certain algorithmic behaviors in a graphical way.

The corresponding book chapter [1] explains the use of these visualization tools and illustrates them with examples arising from practice.

In addition, the package provides functions for computing several quality metrics, such as hypervolume, IGD, IGD+, and epsilon.

Keywords: empirical attainment function, summary attainment surfaces, EAF differences, multi-objective optimization, bi-objective optimization, performance measures, performance assessment, graphical analysis, visualization.

Relevant literature:

1. Manuel López-Ibáñez, Luís Paquete, and Thomas Stützle. Exploratory Analysis of Stochastic Local Search Algorithms in Biobjective Optimization. In T. Bartz-Beielstein, M. Chiarandini, L. Paquete, and M. Preuss, editors, Experimental Methods for the Analysis of Optimization Algorithms, pages 209–222. Springer, Berlin, Germany, 2010.
   (This chapter is also available in a slightly extended form as Technical Report TR/IRIDIA/2009-015).
   [ bibtex | doi: 10.1007/978-3-642-02538-9_9 | Presentation ]

Download and installation

The eaf package is implemented in R. Therefore, a basic knowledge of R is recommended to make use of all features.

The first step before installing the eaf package is to install R. Once R is installed in the system, there are two methods for installing the eaf package:

1. Install within R (automatic download, internet connection required). Invoke R, then

       install.packages("eaf")

2. Download the eaf package from CRAN (you may also need to download and install first the package modeltools), and invoke at the command-line:

       R CMD INSTALL <package>

   where <package> is one of the three versions available: .tar.gz (Unix/BSD/GNU/Linux), .tgz (MacOS X), or .zip (Windows).

Search the R documentation if you need more help to install an R package on your system.

The code for computing the EAF is available as a C program, and it does not require installing R or any R packages. Just download the package source code, uncompress it, and look for the directory src/eaf. The C code can be used to implement your own visualizations instead of the visualizations provided by the eaf package. Compiled executables for computing the EAF can be found under system.file(package="eaf", "bin"). Other useful executable programs can be found there.

The eaf package also contains two Perl scripts that may allow you to generate standard plots without any R knowledge. See inst/scripts/eafplot/ and inst/scripts/eafdiff/ in the package source code. The scripts use the eaf package internally to generate the plots, and, hence, the eaf package must be installed and working.

If you wish to be notified of bugfixes and new versions, please subscribe to the low-volume emo-list, where announcements will be made.

[ Download eaf package from CRAN ] [ Documentation ] [ Development version (GitHub) ]

GitHub (Development version)

If you wish to try the development version, you can install it by executing the following commands within the R console:

    R> install.packages("devtools")
    R> devtools::install_github("MLopez-Ibanez/eaf")

Usage

Once the eaf package is installed, the following R commands will give more information:

    library(eaf)
    ?eaf
    ?eafplot
    ?eafdiffplot
    ?read.data.sets
    example(eafplot)
    example(eafdiffplot) # This one takes some time

Apart from the main R package, the source code contains the following extras in the directory inst/ (after installation, these files can be found at the directory printed by the R command system.file(package="eaf")):

• scripts/eafplot : Perl script to plot summary attainment surfaces.
• scripts/eafdiff : Perl script to plot the differences between the EAFs of two input sets.
• extdata/ : Examples of utilization of the above programs. These are discussed in the corresponding book chapter [1].

In addition, the source code contains the following under src/:

• src/eaf : This C program computes the empirical attainment function in 2 or 3 dimensions. It is NOT required by the other programs, but it is provided as a useful command-line utility. This version is based on the original code written by Carlos M. Fonseca available at http://www.tik.ee.ethz.ch/pisa/. A more recent version is available at Prof. Fonseca's website.
• src/mo-tools : Several tools for working with multi-objective data.

For more information, consult the README files at each subdirectory.

Python

Thanks to rpy2, you can use the eaf package from Python. A complete example would be:

import os
## Uncomment this if you suffer from this bug in cffi 1.13.0
## https://bitbucket.org/rpy2/rpy2/issues/591/runtimeerror-found-a-situation-in-which-we
#os.environ['RPY2_CFFI_MODE'] = "API"

# Tested with rpy2 2.9.2-1 and 3.2.6
import numpy as np
from rpy2.robjects.packages import importr, isinstalled, PackageNotInstalledError
from rpy2.robjects import r as R
from rpy2.robjects import numpy2ri
from rpy2.robjects.vectors import StrVector
numpy2ri.activate()
from rpy2.interactive import process_revents
process_revents.start()

def install_rpackages(packages):
    if not isinstance(packages, list):
        packages = [ packages ]
    utils = importr('utils') # import R's utility package
    # Selectively install what needs to be installed.
    names_to_install = [x for x in packages if not isinstalled(x)]
    if len(names_to_install) > 0:
        print(f"Installing packages: {names_to_install}")
        utils.install_packages(StrVector(names_to_install), repos = "https://cloud.r-project.org", verbose=True)

try:
    eaf = importr("eaf")
except PackageNotInstalledError as e:
    install_rpackages("eaf")
    eaf = importr("eaf") # Retry after install

path = R('system.file(package="eaf", "extdata")')[0] + "/"
alg1 = eaf.read_data_sets_(path + "ALG_1_dat.xz")
alg1 = np.asarray(alg1)
alg2 = np.asarray(eaf.read_data_sets_(path + "ALG_2_dat.xz"))

eaf.eafplot(alg1[:, 0:2], sets=alg1[:,2])

input("Press ENTER to see next plot: ")

eaf.eafdiffplot(alg1, alg2, title_left="A", title_right="B")

License

This software is Copyright (C) 2011-2021 Carlos M. Fonseca, Luís Paquete, Thomas Stützle, Manuel López-Ibáñez and Marco Chiarandini.

This program is free software (software libre); you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

IMPORTANT NOTE: Please be aware that the fact that this program is released as Free Software does not excuse you from scientific propriety, which obligates you to give appropriate credit! If you write a scientific paper describing research that made substantive use of this program, it is your obligation as a scientist to (a) mention the fashion in which this software was used in the Methods section; (b) mention the algorithm in the References section. The appropriate citation is:

• Manuel López-Ibáñez, Luís Paquete, and Thomas Stützle. Exploratory Analysis of Stochastic Local Search Algorithms in Biobjective Optimization. In T. Bartz-Beielstein, M. Chiarandini, L. Paquete, and M. Preuss, editors, Experimental Methods for the Analysis of Optimization Algorithms, pages 209–222. Springer, Berlin, Germany, 2010. doi: 10.1007/978-3-642-02538-9_9

Moreover, as a personal note, I would appreciate it if you would email [email protected] with citations of papers referencing this work so I can mention them to my funding agent and tenure committee.

eaf's Issues

Quick doubts on using the library

Hello,
First of all, congratulations for the eaftools library. It is one of the best tools I've seen to visualize the results of multi-objective optimization. I have three quick questions related to its use (I've searched through the documentation, the website, and the source code and did not find it):
1- Is there a way to extract to which percentile each point on the original dataset would belong (the attainment surfaces), after running the eafplot function for maximizing one criteria and minimizing the other one? Example: I've generated a plot with percentiles = c(0, 20, 40, 60, 80, 100) and I can plot the original points in the dataset on the final plot using extra.points. However, it would be very valuable to know to which "percentile class" each point belonged, so I could calculate the % of points in each "percentile class"
2- Is there an example for running the eaf for 3 dimensions?
3- Are there any visualizations planned for the library for optimization in 3 dimensions?
Thanks a lot,
Roberto

Installation problem

Hi,

I wanted to install this package using devtool::install_github("MLopez-Ibanez/eaf") line, but I got the below error:

Downloading GitHub repo MLopez-Ibanez/eaf@master
from URL https://api.github.com/repos/MLopez-Ibanez/eaf/zipball/master
Installing eaf
"C:/PROGRA~1/R/R-35~1.1/bin/x64/R" --no-site-file --no-environ --no-save  \
  --no-restore --quiet CMD INSTALL  \
  "C:/Users/Amin/AppData/Local/Temp/Rtmp06uYpP/devtools119026131c0f/MLopez-Ibanez-eaf-ea9b50f"  \
  --library="C:/Users/Amin/Documents/R/win-library/3.5" --install-tests 

* installing *source* package 'eaf' ...
** libs

*** arch - i386
c:/Rtools/mingw_32/bin/gcc  -I"C:/PROGRA~1/R/R-35~1.1/include" -DNDEBUG -DVARIANT=4 -DR_PACKAGE -DDEBUG=1         -O3 -Wall  -std=gnu99 -mtune=generic -c Reaf.c -o Reaf.o
In file included from Rcommon.h:6:0,
                 from Reaf.c:1:
common.h:1:1: error: expected identifier or '(' before '.' token
 ../inst/scripts/mo-tools/common.h
 ^
In file included from Reaf.c:2:0:
eaf.h:1:1: error: expected identifier or '(' before '.' token
 ../inst/scripts/eaf/eaf.h
 ^
Reaf.c: In function 'compute_eaf_C':
Reaf.c:67:5: error: unknown type name 'eaf_t'
     eaf_t **eaf = compute_eaf_helper(DATA, nobj, CUMSIZES, nruns, percentile, nlevels);
     ^
Reaf.c:67:5: warning: implicit declaration of function 'compute_eaf_helper' [-Wimplicit-function-declaration]
Reaf.c:67:19: warning: initialization makes pointer from integer without a cast
     eaf_t **eaf = compute_eaf_helper(DATA, nobj, CUMSIZES, nruns, percentile, nlevels);
                   ^
Reaf.c:69:5: warning: implicit declaration of function 'eaf_totalpoints' [-Wimplicit-function-declaration]
     int totalpoints = eaf_totalpoints (eaf, nlevels);
     ^
Reaf.c:78:29: error: request for member 'size' in something not a structure or union
         int npoints = eaf[k]->size;
                             ^
Reaf.c:80:9: warning: implicit declaration of function 'DEBUG2' [-Wimplicit-function-declaration]
         DEBUG2(
         ^
Reaf.c:81:13: error: expected expression before 'int'
             int totalsize = npoints * nobj;
             ^
Reaf.c:89:53: error: request for member 'data' in something not a structure or union
                 rmat[pos + j * totalpoints] = eaf[k]->data[j + i * nobj];
                                                     ^
Reaf.c:94:9: warning: implicit declaration of function 'eaf_delete' [-Wimplicit-function-declaration]
         eaf_delete (eaf[k]);
         ^
Reaf.c: In function 'compute_eafdiff_C':
Reaf.c:114:5: error: unknown type name 'eaf_t'
     eaf_t **eaf = compute_eaf_helper(DATA, nobj, CUMSIZES, nruns, NULL, nruns);
     ^
Reaf.c:114:19: warning: initialization makes pointer from integer without a cast
     eaf_t **eaf = compute_eaf_helper(DATA, nobj, CUMSIZES, nruns, NULL, nruns);
                   ^
Reaf.c:126:29: error: request for member 'size' in something not a structure or union
         int npoints = eaf[k]->size;
                             ^
Reaf.c:130:13: error: expected expression before 'int'
             int totalsize = npoints * nobj;
             ^
Reaf.c:137:53: error: request for member 'data' in something not a structure or union
                 rmat[pos + j * totalpoints] = eaf[k]->data[j + i * nobj];
                                                     ^
Reaf.c:145:29: error: request for member 'size' in something not a structure or union
         int npoints = eaf[k]->size;
                             ^
Reaf.c:149:13: warning: implicit declaration of function 'attained_left_right' [-Wimplicit-function-declaration]
             attained_left_right (eaf[k]->attained + i * eaf[k]->nruns,
             ^
Reaf.c:149:40: error: request for member 'attained' in something not a structure or union
             attained_left_right (eaf[k]->attained + i * eaf[k]->nruns,
                                        ^
Reaf.c:149:63: error: request for member 'nruns' in something not a structure or union
             attained_left_right (eaf[k]->attained + i * eaf[k]->nruns,
                                                               ^
Reaf.c: In function 'polygon_len':
Reaf.c:165:20: error: 'objective_MIN' undeclared (first use in this function)
     while (*src != objective_MIN)
                    ^
Reaf.c:165:20: note: each undeclared identifier is reported only once for each function it appears in
Reaf.c: In function 'polygon_copy':
Reaf.c:174:20: error: 'objective_MIN' undeclared (first use in this function)
     while (*src != objective_MIN) {
                    ^
Reaf.c: In function 'compute_eafdiff_area_C':
Reaf.c:199:5: error: unknown type name 'eaf_t'
     eaf_t **eaf = compute_eaf_helper(DATA, nobj, CUMSIZES, nruns, NULL, nruns);
     ^
Reaf.c:199:19: warning: initialization makes pointer from integer without a cast
     eaf_t **eaf = compute_eaf_helper(DATA, nobj, CUMSIZES, nruns, NULL, nruns);
                   ^
Reaf.c:201:5: error: unknown type name 'eaf_polygon_t'
     eaf_polygon_t *p = eaf_compute_area (eaf, nruns);
     ^
Reaf.c:201:5: warning: implicit declaration of function 'eaf_compute_area' [-Wimplicit-function-declaration]
Reaf.c:201:24: warning: initialization makes pointer from integer without a cast
     eaf_polygon_t *p = eaf_compute_area (eaf, nruns);
                        ^
Reaf.c:209:5: warning: implicit declaration of function 'vector_int_size' [-Wimplicit-function-declaration]
     int ncol = vector_int_size(&p->col);
     ^
Reaf.c:209:34: error: request for member 'col' in something not a structure or union
     int ncol = vector_int_size(&p->col);
                                  ^
Reaf.c:211:5: error: invalid use of void expression
     DEBUG2(Rprintf ("ncol: %d\n", ncol));
     ^
Reaf.c:218:5: warning: implicit declaration of function 'vector_objective_begin' [-Wimplicit-function-declaration]
     double * p_xy = vector_objective_begin(&p->xy);
     ^
Reaf.c:218:46: error: request for member 'xy' in something not a structure or union
     double * p_xy = vector_objective_begin(&p->xy);
                                              ^
Reaf.c:220:9: warning: implicit declaration of function 'vector_int_at' [-Wimplicit-function-declaration]
         double color = vector_int_at(&p->col, k);
         ^
Reaf.c:220:40: error: request for member 'col' in something not a structure or union
         double color = vector_int_at(&p->col, k);
                                        ^
Reaf.c:224:9: error: invalid use of void expression
         DEBUG2(Rprintf ("color: %d, len = %d\n", color, len));
         ^
Reaf.c:232:9: warning: implicit declaration of function 'vector_int_set' [-Wimplicit-function-declaration]
         vector_int_set(&p->col, k, color);
         ^
Reaf.c:232:26: error: request for member 'col' in something not a structure or union
         vector_int_set(&p->col, k, color);
                          ^
Reaf.c:236:21: error: invalid use of void expression
                     left_len, right_len, left_ncol, right_ncol));
                     ^
Reaf.c:248:37: error: request for member 'xy' in something not a structure or union
     p_xy = vector_objective_begin(&p->xy);
                                     ^
Reaf.c:253:37: error: request for member 'col' in something not a structure or union
         int color = vector_int_at(&p->col, k);
                                     ^
Reaf.c:267:5: warning: implicit declaration of function 'vector_int_dtor' [-Wimplicit-function-declaration]
     vector_int_dtor (&p->col);
     ^
Reaf.c:267:24: error: request for member 'col' in something not a structure or union
     vector_int_dtor (&p->col);
                        ^
Reaf.c:268:5: warning: implicit declaration of function 'vector_objective_dtor' [-Wimplicit-function-declaration]
     vector_objective_dtor (&p->xy);
     ^
Reaf.c:268:30: error: request for member 'xy' in something not a structure or union
     vector_objective_dtor (&p->xy);
                              ^
Reaf.c: In function 'read_data_sets':
Reaf.c:294:5: error: unknown type name 'objective_t'
     objective_t *data = NULL;
     ^
Reaf.c:300:5: warning: implicit declaration of function 'read_objective_t_data' [-Wimplicit-function-declaration]
     read_objective_t_data (filename, &data, &nobj, &cumsizes, &nruns);
     ^
Reaf.c:307:30: warning: passing argument 2 of 'double_transpose' from incompatible pointer type
     double_transpose (rdata, data, ntotal, nobj);
                              ^
In file included from Reaf.c:1:0:
Rcommon.h:104:1: note: expected 'const double *' but argument is of type 'int *'
 double_transpose(double *dst, const double *src,
 ^
Reaf.c:309:15: warning: unused variable 'i' [-Wunused-variable]
     int k, j, i;
               ^
make: *** [Reaf.o] Error 1
ERROR: compilation failed for package 'eaf'
* removing 'C:/Users/Amin/Documents/R/win-library/3.5/eaf'
* restoring previous 'C:/Users/Amin/Documents/R/win-library/3.5/eaf'
In R CMD INSTALL
Installation failed: Command failed (1)

Do you have any idea what is this error for?
Thanks.

ValueError with numpy > 1.16.4 in README Python example

The example in the README of running eaf with Python requires numpy<=1.16.4 (and no later, until rpy2 v3.0.x is released and may fix it). Note that this is referenced in another issue for rpy2 (conda-forge/rpy2-feedstock#28).

Better default reference point for HV calculations

Use the method suggested here: https://dl.acm.org/doi/10.1162/evco_a_00226
to define a default reference point.