dme65 / pysot

Surrogate Optimization Toolbox for Python

pySOT: Python Surrogate Optimization Toolbox

The Python Surrogate Optimization Toolbox (pySOT) is an asynchronous parallel optimization toolbox for computationally expensive global optimization problems. pySOT is built on top of the Plumbing for Optimization with Asynchronous Parallelism (POAP), which is an event-driven framework for building and combining asynchronous optimization strategies. POAP has support for both threads and MPI.

pySOT implements many popular surrogate optimization algorithms such as the Stochastic RBF (SRBF) and DYCORS methods by Regis and Shoemaker, and the SOP method by Krityakierne et. al. We also support Expected Improvement (EI) and Lower Confidence Bounds (LCB), which are popular in Bayesian optimization. All optimization algorithms can be used in serial, synchronous parallel, and asynchronous parallel and we support both continuous and integer variables.

The toolbox is hosted on GitHub: https://github.com/dme65/pySOT

Documentation: http://pysot.readthedocs.io/

Installation

Installation instructions are available at: http://pysot.readthedocs.io/en/latest/quickstart.html

Examples

Several pySOT examples and notebooks can be found at:

https://github.com/dme65/pySOT/tree/master/examples

https://github.com/dme65/pySOT/tree/master/notebooks

Citing Us

If you use pySOT, please cite the following paper: David Eriksson, David Bindel, Christine A. Shoemaker. pySOT and POAP: An event-driven asynchronous framework for surrogate optimization. arXiv preprint arXiv:1908.00420, 2019

@article{eriksson2019pysot,
  title={pySOT and POAP: An event-driven asynchronous framework for surrogate optimization},
  author={Eriksson, David and Bindel, David and Shoemaker, Christine A},
  journal={arXiv preprint arXiv:1908.00420},
  year={2019}
}

FAQ

Q: Can I use pySOT with MPI?
A: Yes. You need to install mpi4py in order to use the MPIController in POAP.

Q: I used pySOT for my research and want to cite it
A: Please cite our preprint which is currently under review!

Q: Is there support for Python 2?
A: Python 2 support was removed in version 0.2.0

Q: I can't find the MARS interpolant
A: You need to install py-earth in order to use MARS. More information is available here: https://github.com/scikit-learn-contrib/py-earth

pysot's Issues

MIssing parentheses

In surrogate_strategy.py I believe there is supposed to be a parenthesis on line 163 otherwise it never catches the error of having less evaluation points than the minimum. Currently in the example problems I can put any value for max_evals when calling strategy and the error won't be thrown.

Stopping criteria based on objective and/or decision variable tolerance

Hi all, is this available in pySOT already? From searching in the documentation and previous issues, I could only find two stopping criteria, one based on the maximum number of function evaluations and another on maximum run time.

I would like to stop the search when both my objective and decision variables (norm) do not change (within a tolerance) from one iteration to the next. In my case, sometimes the solver converges within 10 evaluations, other times it takes 20 evaluations, and in the former I would like to obtain an answer after those 10 evaluations since the remaining 10 evaluations may take up to a day to complete.

Thank you.

"python setup.py install" is not working

Problem 1: cannot install inspyred because of missing README.md file.
Error log:

Processing dependencies for pySOT==0.1.18
Searching for inspyred
Reading https://pypi.python.org/simple/inspyred/
Best match: inspyred 1.0.1
Downloading https://pypi.python.org/packages/source/i/inspyred/inspyred-1.0.1.tar.gz#md5=ce51b599f5682ec526c97412f3ebaf45
Processing inspyred-1.0.1.tar.gz
Writing /tmp/easy_install-3cWNNt/inspyred-1.0.1/setup.cfg
Running inspyred-1.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3cWNNt/inspyred-1.0.1/egg-dist-tmp-_n29fg
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.14.tar.gz
Extracting in /tmp/easy_install-3cWNNt/inspyred-1.0.1/temp/tmpD3lh4S
Now working in /tmp/easy_install-3cWNNt/inspyred-1.0.1/temp/tmpD3lh4S/distribute-0.6.14
Building a Distribute egg in /tmp/easy_install-3cWNNt/inspyred-1.0.1
/tmp/easy_install-3cWNNt/inspyred-1.0.1/distribute-0.6.14-py2.7.egg
error: [Errno 2] No such file or directory: '/tmp/easy_install-3cWNNt/inspyred-1.0.1/README.md'

After manually installing inspyred and removing the need for README.md by editing inspyred's setup.cfg I have this problem:

Problem 2:

Processing dependencies for pySOT==0.1.18
Traceback (most recent call last):
  File "setup.py", line 15, in <module>
    'Programming Language :: Python :: 2.7',
  File "/home/ilija/anaconda2/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/home/ilija/anaconda2/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/home/ilija/anaconda2/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/install.py", line 73, in run
    self.do_egg_install()
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/install.py", line 101, in do_egg_install
    cmd.run()
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 342, in run
    self.easy_install(spec, not self.no_deps)
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 562, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 614, in install_item
    self.process_distribution(spec, dist, deps)
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 666, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 557, in resolve
    requirements.extend(dist.requires(req.extras)[::-1])
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 2172, in requires
    dm = self._dep_map
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 2164, in _dep_map
    for extra,reqs in split_sections(self._get_metadata(name)):
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 2628, in split_sections
    for line in yield_lines(s):
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 1854, in yield_lines
    for ss in strs:
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 2186, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 1174, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 1166, in get_metadata
    return self._get(self._fn(self.egg_info,name))
  File "/home/ilija/anaconda2/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 1231, in _get
    return self.loader.get_data(path)
zipimport.ZipImportError: bad local file header in /home/ilija/anaconda2/lib/python2.7/site-packages/pySOT-0.1.18-py2.7.egg

So I gave up, and installed via pip install pySOT

pySOT is slow to generate new points with large dataset

I am using pySOT with EIStrategy and GPRegressor surrogate model for a 6-dimensional optimization problem (all axes are continuous). I am using 10 worker threads. I have Python 3.11.1 and Windows 10.

In recent runs, I have observed that after a little more than 1000 data points, generating new points starts to slow WAY down. Before this occurs, most time is spent evaluating the objective function, with worker threads being assigned a new task within seconds (or ms even) of finishing the previous evaluation. However, at a little over 1000 data points, I noticed that almost all the time is spent waiting for new assignments.

This is confirmed by computing resource allocation. Prior to 1k, the evaluations called by worker threads take up all available CPU. After 1k, Python takes ~30% and worker threads rarely take any at all. In fact, only one worker thread is ever actually evaluating the objective at once because it gets done evaluating the objective before the next point has been generated.

This leads me to suspect that the process of generating new points slows WAY down around 1k points.

1. Is this expected behavior? It is totally possible I've got a bug in my code somewhere that is causing this.
2. Any tips to work around this?
   a. I would be happy to accept generating less optimal points if it meant more could be generated faster.
   b. I have some capacity to rewrite and/or multithread functions in C, if there is a specific function that may be the bottleneck.

Edit: this SE article seems relevant. https://stats.stackexchange.com/questions/326446/gaussian-process-regression-for-large-datasets

Installing from source doesn't work

Following the latest documentation, I try to install it like this:

virtualenv .venv
. .venv/bin/activate
git clone https://github.com/dme65/pySOT
cd pySOT
pip install inspyred
python setup.py install

I get this error:
ImportError: No module named numpy.distutils.core

Running python setup.py install again, for the second time, will complete the installation.

What's the algorithm complexity of DYCORS with CubicRBF?

Thanks

Ability to save the state of the program after n function evalutions

It would be very useful if it's possible to save the state of the program after every function evaluation. One function evaluation can take several hours, so server crash after 200 or something function evaluation is very painful 😅

Add RBF regression surfaces

Getting stuck after raised ValueError.

When a function evaluation doesn't complete (error in the external objective function or some other issues), I raise ValueError. However, after raising ValueError pySOT doesn't continue. I assume is getting stuck in some infinite loop, since the process continue to use the CPU 100%. I left it for several hours and didn't change.

To reproduce follow these steps:

virtualenv venv_test
. ./venv_test/bin/activate
pip install numpy
pip install inspyred
git clone [email protected]:dme65/pySOT.git
cd pySOT/
# edit the Ackley class so it raises ValueError
vim pySOT/test/Ackley.py   
python setup.py install
# edit test_simple.py so it uses the new Ackley
vim pySOT/test/test_simple.py
cd pySOT/test/
python test_simple.py

Output:

(venv_test)ilija@deep03:~/for_test/pySOT/pySOT/test$ python test_simple.py

Number of threads: 1
Maximum number of evaluations: 1000
Search strategy: CandidateDYCORS
Experimental design: Latin Hypercube
Ensemble surrogates: Cubic RBF, domain scaled to unit box
10-dimensional Ackley function
Global optimum: f(0,0,...,0) = 0
[-12.61363636  -4.65909091   6.47727273  -9.43181818   9.65909091
 -12.61363636   8.06818182  -4.65909091  -6.25        16.02272727]
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "build/bdist.linux-x86_64/egg/poap/controller.py", line 426, in run
    self.handle_eval(record)
  File "build/bdist.linux-x86_64/egg/poap/controller.py", line 453, in handle_eval
    self.finish_success(record, self.objective(*record.params))
  File "/home/ilija/for_test/pySOT/pySOT/test/Ackley.py", line 19, in objfunction
    raise ValueError('Dimension mismatch')
ValueError: Dimension mismatch

I have attached the modified Ackley.py and test_simple.py.

Archive.zip

`examples` and `tests` are being installed as top-level packages (directly in site-packages)

When installing pySOT with pip, I think that examples and tests were supposed to be inside the pySOT package, but they are installed in a directory outside of it.

PendingDeprecationWarning from use of NumPy matrices

We recently tracked down a PendingDeprecationWarning from our application test suite to the uses of np.asmatrix in this repository. I can't post the actual traceback here, but here's an example of the same effect (Python 3.7.2, NumPy 1.15.4):

mdickinson$ python -Wd
Python 3.7.2 (default, Dec 30 2018, 08:55:50) 
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.__version__
'1.15.4'
>>> np.asmatrix(np.ones((3, 3)))
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/matrixlib/defmatrix.py:68: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray.
  return matrix(data, dtype=dtype, copy=False)
matrix([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]])

For convenience, the message text is:

PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray.

Problem with too many failed function evaluations.

The toolbox evaluates the same function evaluations over and over again, when they failed to execute, thus getting stuck in a infinite loop. See #6 and #8.

MPI example gets stuck

Hi, I am trying the examples provided and the MPI example gets stuck, not error, just not advancing nor calling the objective function. I have tried the same code under windows 10 (after installing MPI from microsoft) and under Ubuntu 18.

I wonder is this is a bug, of if I am missing something.

The code is this:

"""
.. module:: mpiexample_simple_mpi
  :synopsis: Simple MPI example
.. moduleauthor:: David Eriksson <[email protected]>
"""

from pySOT.experimental_design import SymmetricLatinHypercube
from pySOT.strategy import SRBFStrategy
from pySOT.surrogate import RBFInterpolant, CubicKernel, LinearTail
from pySOT.optimization_problems import OptimizationProblem

from poap.mpiserve import MPIController, MPISimpleWorker
import numpy as np
import os.path
import logging

# Try to import mpi4py
try:
    from mpi4py import MPI
except Exception as err:
    print("ERROR: You need mpi4py to use the POAP MPI controller.")
    exit()


class Ackley(OptimizationProblem):
    """
    Ackley function

    .. math::
        f(x_1,\\ldots,x_n) = -20\\exp\\left( -0.2 \\sqrt{\\frac{1}{n} \
        \\sum_{j=1}^n x_j^2} \\right) -\\exp \\left( \\frac{1}{n} \
        \\sum{j=1}^n \\cos(2 \\pi x_j) \\right) + 20 - e

    subject to

    .. math::
        -15 \\leq x_i \\leq 20

    Global optimum: :math:`f(0,0,...,0)=0`

    :ivar dim: Number of dimensions
    :ivar lb: Lower variable bounds
    :ivar ub: Upper variable bounds
    :ivar int_var: Integer variables
    :ivar cont_var: Continuous variables
    :ivar min: Global minimum value
    :ivar minimum: Global minimizer
    :ivar info: String with problem info
    """
    def __init__(self, dim=10):
        self.dim = dim
        self.min = 0
        self.minimum = np.zeros(dim)
        self.lb = -15 * np.ones(dim)
        self.ub = 20 * np.ones(dim)
        self.int_var = np.array([])
        self.cont_var = np.arange(0, dim)
        self.info = str(dim) + "-dimensional Ackley function \n" + "Global optimum: f(0,0,...,0) = 0"

    def eval(self, x):
        """
        Evaluate the Ackley function  at x

        :param x: Data point
        :type x: numpy.array
        :return: Value at x
        :rtype: float
        """
        print('eval!')
        self.__check_input__(x)
        d = float(self.dim)
        return -20.0 * np.exp(-0.2*np.sqrt(np.sum(x**2) / d)) - np.exp(np.sum(np.cos(2.0*np.pi*x)) / d) + 20 + np.exp(1)


def main_worker(objfunction):
    MPISimpleWorker(objfunction).run()


def main_master(opt_prob, num_workers):
    if not os.path.exists("./logfiles"):
        os.makedirs("logfiles")
    if os.path.exists("./logfiles/mpiexample_mpi.log"):
        os.remove("./logfiles/mpiexample_mpi.log")
    logging.basicConfig(filename="./logfiles/mpiexample_mpi.log",
                        level=logging.INFO)

    max_evals = 500

    rbf = RBFInterpolant(dim=opt_prob.dim,
                         kernel=CubicKernel(),
                         tail=LinearTail(opt_prob.dim))

    slhd = SymmetricLatinHypercube(dim=opt_prob.dim,
                                   num_pts=2*(opt_prob.dim+1))

    # Create a strategy and a controller
    strategy = SRBFStrategy(max_evals=max_evals,
                            opt_prob=opt_prob,
                            exp_design=slhd,
                            surrogate=rbf,
                            asynchronous=True,
                            batch_size=num_workers)
    controller = MPIController(strategy)

    print("Number of workers: {}".format(num_workers))
    print("Maximum number of evaluations: {}".format(max_evals))
    print("Strategy: {}".format(controller.strategy.__class__.__name__))
    print("Experimental design: {}".format(slhd.__class__.__name__))
    print("Surrogate: {}".format(rbf.__class__.__name__))

    print('Running...')
    result = controller.run()
    print('Best value found: {0}'.format(result.value))
    print('Best solution found: {0}\n'.format(np.array_str(result.params[0],
                                                           max_line_width=np.inf,
                                                           precision=5,
                                                           suppress_small=True)))


def mpi_example_simple():

    # Optimization problem
    ackley = Ackley(dim=10)

    # Extract the rank
    comm = MPI.COMM_WORLD
    rank = comm.Get_rank()
    nprocs = comm.Get_size()

    if rank == 0:
        main_master(ackley, nprocs)
    else:
        main_worker(ackley.eval)


if __name__ == '__main__':
    mpi_example_simple()

pySOT repeats the same function evaluations when using external C++ objective function

I'm using pySOT as the example 11.6 external C++ objective function as shown here https://github.com/dme65/pySOT/blob/444070e31887525ed418166d9107b6a355353fc9/docs/pySOT.pdf.

When I set the number of evaluations to 2 for example, it will repeat those same 2 over and over again. It would never finish.

If needed I can share the code privately.

Full log:

(.venv)ilija@deep03:~/h$ python pySOT_runner.py 1 2

Number of threads: 1
Maximum number of evaluations: 2
Search strategy: Candidate DyCORS
Experimental design: Latin Hypercube
Surrogate: Cubic RBF
{
  epoch : 17
  showProgress : false
  seed : 139
  hiddenNodes : 130
  experimentId : 1
  threads : 16
  momentum : 0.8071425
  deviceId : 1
  batchSize : 128
  learnRate : 0.05075
  plot : false
  save : "logs"
  mean : 0.002500075
  std : 0.00453571428571
  learnRateDecay : 0
}
0.1937

{
  epoch : 19
  showProgress : false
  seed : 139
  hiddenNodes : 173
  experimentId : 1
  threads : 16
  momentum : 0.8214275
  deviceId : 1
  batchSize : 128
  learnRate : 0.0223214285714
  plot : false
  save : "logs"
  mean : 0.000357239285714
  std : 0.0893928571429
  learnRateDecay : 0
}
0.6733

{
  epoch : 17
  showProgress : false
  seed : 139
  hiddenNodes : 130
  experimentId : 1
  threads : 16
  momentum : 0.8071425
  deviceId : 1
  batchSize : 128
  learnRate : 0.05075
  plot : false
  save : "logs"
  mean : 0.002500075
  std : 0.00453571428571
  learnRateDecay : 0
}
0.1937

{
  epoch : 19
  showProgress : false
  seed : 139
  hiddenNodes : 173
  experimentId : 1
  threads : 16
  momentum : 0.8214275
  deviceId : 1
  batchSize : 128
  learnRate : 0.0223214285714
  plot : false
  save : "logs"
  mean : 0.000357239285714
  std : 0.0893928571429
  learnRateDecay : 0
}
0.6733

And it will repeat this forever.
These are the parameters to optimise

{
  epoch : 19
  showProgress : false
  seed : 139
  hiddenNodes : 173
  experimentId : 1
  threads : 16
  momentum : 0.8214275
  deviceId : 1
  batchSize : 128
  learnRate : 0.0223214285714
  plot : false
  save : "logs"
  mean : 0.000357239285714
  std : 0.0893928571429
  learnRateDecay : 0
}

and 0.6733 is the function evaluation result. You can see it repeats the same two function evaluations, i.e. it doesn't try with different set of parameters.

Missing ez_setup requirement

The module ez_setup is also required (https://github.com/dme65/pySOT/blob/master/setup.py#L13)

Consider improving the parameter configuration interface

Currently the parameter configuration is being set in this way:

        self.xlow = -15 * np.ones(dim)
        self.xup = 20 * np.ones(dim)
        self.integer = []
        self.continuous = np.arange(0, dim)

This is very cumbersome interface, especially when you have 15 parameters, each with different ranges, and are of different type. It's very easy to make a mistake.
Further if the type of parameter is left unassigned, there is no error or warning. Example

      self.integer = [0, 1, 2]
      self.continuous = [4, 5]

Here the fourth parameter is left without a type (integer or continuous), but there is no error or warning produced, and the algorithm just considers it as continuous.

Also, it would be useful if we can assign a name to a parameter.

I suggest to adopt some JSON configuration file, similar to what other optimization algorithms use. Example:

       {
           parameters:[
                      {
                          name: "learning_rate",
                          low: 10,
                          high: 20,
                          type: "Integer"
                      },
                     ...
           ]
       }

Most importantly produce an error or at least a warning if not all of the configuration properties are set.

Thanks

Missing requirements.txt

The file requirements.txt is missing. It is referenced here:
https://github.com/dme65/pySOT/blob/master/docs/pySOT.tex#L383

SurrogateUnitBox not longer required for RBF?

Just saw that my new docker image pulled your newest version. Normally i used to wrap the RBFInterpolant into the SurrogateUnitBox(model=RBFInterpolant()). Now SurrogateUnitBox doesn't exist anymore. Before my models where super bad without. Did you fix it now and it works without?

Thanks for your help
Cheers
Fabi

stopping criteria and failed evaluation question

Hi there,
to be sure, currently the only one stopping criteria supported in pySOT is maximum number of evaluations, am I right?
The second question is about failed evaluation. What should my own "objfunction" method return to consider it as failed evaluation by pySOT? Should "ValueError" exception arise? And what about optimization process behavior in this case, does it stop or continue? (I've read the issues #8 and #9)

Add back option to restart algorithms when they converge early

pySOT automatically restarted before version 0.2.0, but this option was removed to allow users to look at the results before deciding to restart. We should add back an option for the strategies to restart if it's desired by the user to do all function evaluations without looking at any intermediate results.

Possible bug in Keane test script

Hi,

In the eval_ineq_constraints() method in Keane.py, it seems to me that the code

vec = np.zeros((x.shape[0], 2))
vec[:, 0] = 0.75 - np.prod(x)
vec[:, 1] = np.sum(x) - 7.5 * self.dim

should be

vec = np.zeros((a.shape[0], 2))
vec[:, 0] = 0.75 - np.prod(a, axis=1)
vec[:, 1] = np.sum(a, axis=1) - 7.5 * self.dim

I am basing my understanding on the documentation on the expected form of eval_ineq_constraints() method explained here.

Evaluates the uncertainty of the surrogate model at points x

Hello,

Could you please give me an example that prints out "the uncertainty of the surrogate model at points x" as said in the online manual?

predict_std(x): Evaluates the uncertainty of the surrogate model at points x

I am using your code to calibrate a highly non-linear groundwater model with a high number of model parameters (large than 40). The code worked perfectly and saved me a lot of time. I looking forward to see if I can use the code for uncertainty analysis related to estimated model parameters. I am using pysot with DYCORStrategy.

Thanks,
Hai.

Multiple threads does not work

Hi,

I run a simple example with num_threads = 1, it is OK but with
multiple threads=4, I got the following error. I use the same example code (example_mars()).
However I replace the objective function as:

import pySOT.optimization_problems as sprob
PORT=8500
url="http://localhost:" + str(PORT) + "pysot_test"

class PySOTTestProblem(sprob.OptimizationProblem):
def init(self):
self.dim=1
self.info = "PySOT test Problem" #optional
self.lb = 0.9np.array([0.0])
self.ub = 1.1np.array([2*np.pi])
self.int_var = np.array([])
self.cont_var = np.arange(0, 1)

def eval(self, x):
    """Evaluate the objective function at x
    
    :param x: Data point
    :type x: numpy.array
    :return: Value at x
    :rtype: float
    """
    self.__check_input__(x)       
    print("x=", x)
    spec=dict(x=x[0])
    results = requests.post(url,json=spec, timeout=6400).json()
    objective =  float(results['objective'])
    print("x=", beta, "objective=", objective)
    return objective

==============================================
on the server side
I have a web server (gunicorn and falcon) running and return the value of np.sin(x).

###========begin of error===========================

ValueError Traceback (most recent call last)
in
----> 1 equity_model_adaptation()

in equity_model_adaptation()
38
39 # Run the optimization strategy
---> 40 result = controller.run()
41
42 print('Best value found: {0}'.format(result.value))

~/miniconda3/envs/dp36/lib/python3.6/site-packages/poap/controller.py in run(self, merit, filter)
342 """
343 try:
--> 344 return self._run(merit=merit, filter=filter)
345 finally:
346 self.call_term_callbacks()

~/miniconda3/envs/dp36/lib/python3.6/site-packages/poap/controller.py in _run(self, merit, filter)
312 self._run_queued_messages()
313 time.sleep(0) # Yields to other threads
--> 314 proposal = self.strategy.propose_action()
315 if not proposal:
316 self._run_message()

~/miniconda3/envs/dp36/lib/python3.6/site-packages/pySOT/strategy.py in propose_action(self)
279 self.phase = 2
280 if self.asynchronous: # Always make proposal with asynchrony
--> 281 self.generate_evals(num_pts=1)
282 elif self.pending_evals == 0: # Make sure the entire batch is done
283 self.generate_evals(num_pts=self.batch_size)

~/miniconda3/envs/dp36/lib/python3.6/site-packages/pySOT/strategy.py in generate_evals(self, num_pts)
562 opt_prob=self.opt_prob, num_pts=num_pts, surrogate=self.surrogate,
563 X=self.X, fX=self.fX, Xpend=self.Xpend, weights=weights,
--> 564 sampling_radius=self.sampling_radius, num_cand=self.num_cand)
565
566 for i in range(num_pts):

~/miniconda3/envs/dp36/lib/python3.6/site-packages/pySOT/auxiliary_problems.py in candidate_srbf(num_pts, opt_prob, surrogate, X, fX, weights, Xpend, sampling_radius, subset, dtol, num_cand)
106 """
107 # Find best solution
--> 108 xbest = np.copy(X[np.argmin(fX), :]).ravel()
109
110 # Fix default values

~/miniconda3/envs/dp36/lib/python3.6/site-packages/numpy/core/fromnumeric.py in argmin(a, axis, out)
1099
1100 """
-> 1101 return _wrapfunc(a, 'argmin', axis=axis, out=out)
1102
1103

~/miniconda3/envs/dp36/lib/python3.6/site-packages/numpy/core/fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
49 def _wrapfunc(obj, method, *args, **kwds):
50 try:
---> 51 return getattr(obj, method)(*args, **kwds)
52
53 # An AttributeError occurs if the object does not have

ValueError: attempt to get argmin of an empty sequence
==========end of error========================================

pySOT results are not consistent with the DYCORS paper

pySOT performs significantly worse on 200-dimensional problems than was reported for the DYCORS method in "Combining radial basis function surrogates and dynamic coordinate search in high-dimensional expensive black-box optimization", Regis and Shoemaker.

Dependency on pyDOE should be updated to work with pyDOE2

pyDOE hasn't been released for more than a year and it seems to be no longer actively maintained. I suggest changing the dependency so that users of pySOT may use the framework without any difficulties. As to the date, I am able to use the pySOT.experimental_design.TwoFactorial methods only if I change the pyDOE's file manually, which turns out to make the deployment of any applications more difficult.

pySOT/pySOT/experimental_design.py

Line 14 in c3f2fce

import pyDOE as pydoe

Using known objective function values

Hi,
since I'm not very experienced in surrogate-based optimization I'd like to ask whether there is in pySOT some possibility to use points with already known objective function values before starting optimization itself? Is it possible to do it by own implementation of experimental design? But what are the most important properties which that experimental of design has to fulfil? (I’ve just read some information in online documentation.)
Note: My question is motivated by my optimization task. I need to optimize SW (image processing) module where SW efficiency (objective function value) is possible to get by setting parameters and running time-consuming simulation only. But I have already had some parameters settings, which could be used as "initial" point. Is it good idea? Moreover, there are some other known points as a result of previous testing which may be possibly useful as well (?).
Radek

Add a new Kriging (or GP) module

Use of "extra" and "extra_vals"

Hi,

I am trying to understand the role of setting the extra and extra_vals parameters in something like SRBFStrategy(). It seems like if I append the controller's fevalslist with appropriately populated EvalRecord objects (as shown in this example), the controller is able to leverage this prior information. What is purpose of also setting the extra, extra_vals parameters in SRBFStrategy()?

Regards

Implement SO-MI

We should implement a strategy that is tailored for mixed-integer problems, such as SO-MI (A surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems, Müller, Juliane; Shoemaker, Christine; Piche, Robert)

In gp.py, predict() returns incorrectly-sized array, causing IndexError exception in ei_merit().

In gp.py, the predict() method returns an array of size (num_pts,). This does not match the returned size of the predict_std() method, which instead returns an array of size (num_pts,1).

This causes issues in ei_merit.py, in the ei_merit() function. When mu and sig are assigned different sizes on line 23, gamma (and subsequently beta then ei) will become a square array of size (num_pts, num_pts). This causes an IndexError exception on line 31, when the incorrectly-sized ei is indexed by the correctly-sized dmerit.

Potential solution:
In gp.py, in the predict() method, change line 71 to more closely match the form of line 85. Suggestion below:
return self._mu + self._sigma * np.expand_dims(self.model.predict(xx), axis=1)

Note: this issue appeared suddenly for me, without any updates to pySot itself, so it is possible that either:

1. I introduced this issue by passing in bad values somewhere. However, the proposed fix resolved my issue, so I would say it's not likely that that is the problem.
2. While I did not track the timing closely, there is a chance that this issue began to occur after upgrading from python 3.7 from 3.11.

pySOT may propose the same point twice on pure integer problems

add_points() does not handle objective functions that output integers

In surrogate.py, in the add_points() method, an AttributeError exception is raised by calling fx.ndim (line 64) when fx is an integer.

Potential solutions:

1. On line 62, expand type-checking to include integers as well as floats and update comment on line 59 accordingly.
2. Update documentation here to specify that the eval method must return a float. The comments inside add_points() already reflect this, but the documentation on readthedocs.io does not.

comparisons

PySOT doing very well here versus Hyperopt, Optuna etc...
https://www.microprediction.com/blog/optimize

Documentation has an error

Hey guys, for the ub and lb you declare it as a function. But when you pass it later to unit box you declare it in unit box function docu as a np.array. I think the func should just be replaced by np.array

:type lb: function
:param ub: Upper variable bounds, of size 1 x dim
:type ub: function

Cheers :)

Comments in ei_merit() do not match the actual sizes of parameters passed to the function

In ei_merit.py, in the ei_merit() method, the sizes of the parameters passed do not match the comments.

My setup:

• 6 dimensional optimization problem (all dimensions are continuous)
• EIStrategy
• SymmetricLatinHypercube (num_pts = 2*dim + 1)
• PolyRegressor

To reproduce: put a breakpoint on line 23 in ei_merit() and use the debug console to observe the following:

np.shape(X)
(100, 6)
np.shape(fX)
(6, 1)
np.shape(XX)
(18, 6)

Two semi-separate issues:

1. The comment on line 15 states that parameter XX should have shape m x 1. It should actually by m x dim since XX represents inputs to the objective function, which have size equal to the dimension of the problem.
   I think this should be an easy fix, assuming that I diagnosed the issue correctly.

2. The comments on lines 13 and 15 collectively state that the dimensions on axis 0 of fX and XX should match (and be equal to m). While this would make sense to me, I have not observed that in practice (see console output above). I believe this discrepancy can be traced back to line 33 of ei_ga.py, where the pending evaluations are appended to the previously evaluated points before later passing them to ei_merit on line 41. This makes the array of previously evaluated points larger than the array of values at previously evaluated points, by an amount equal to the number of pending evaluations.
   I am not sure if I correctly diagnosed the root cause of this issue. While the comment is definitely wrong in the current state, I do not know whether the comment or the code meets the intent of the design. As far as I can tell, the code in its current state does not cause any problems with effectiveness or reliability.

how to surrogate model with multiple variable?

I have the three parameters (node1,node2,node2) with each parameters I have 20 values.
nodes1 = np.array([0.05675, 0.05934, 0.05633, 0.0557 , 0.05702, 0.06401, 0.06322, 0.06571, 0.06099, 0.05832, 0.06196, 0.06463, 0.05507, 0.06351, 0.06287, 0.06122, 0.05407, 0.05985, 0.05774,0.06015])

nodes2 = np.array([0.9486, 0.9095, 0.9856, 0.9318, 1.0477, 1.0489,1.0663, 0.9184, 0.9646, 1.0345, 1.0168, 1.0565, 0.9727, 0.9907, 0.9277, 0.9548, 1.0933, 1.0751,1.0026, 1.0231])

nodes3 = np.array([51.813, 54.279, 52.659, 51.197 , 46.629, 49.791, 48.581, 54.799, 46.413, 47.078, 52.367, 48.204, 50.389, 45.402, 47.893, 50.796 , 49.332, 53.323, 53.713, 45.757])

with a group parameter of input I have a output values. So I have the 20 output values
example: (node1_1,node2_1,node3_1)=(0.05675,0.9486,51.813) I have a output values = 0.0204232

output=np.array([0.0204232,0.0205054,0.0204971,0.0204463,0.0206686,0.0206678,0.0206883,0.0204627,0.020426,0.0206532,0.0206322,0.020677,0.0204431,0.0205319,0.0204508,0.0204115,0.020721,0.0206988,0.0206179,0.0206418])

How to estimate the surrogate model between (node1,node2,node3) with output? and how to know the detail equation of the surrogate model?

please help me, thanks you

mpiexample_simple.py fails after the increase of number of workers

I am trying to run the script using this command:

mpiexec -n 20 python mpi_example.py

Unfortunately, it fails with AssertionError. And it works just fine with 10 workers (example below.)
Could you please explain what has happened?

Evaluating expensive blackbox constraints?

Hi,

Is it possible to use pySOT with expensive blackbox constraints?

For example, if the objfunction() method in the optimisation problem class can be written to use an external piece of simulation software, then how might this be applied for the eval_ineq_constraints() method?

In my case, I would be interpreting simulation results from an external program to provide objective function values and constraint function values to pySOT. But the eval_ineq_constraints() method seems to accept a numpy array for a number of different points all at once, unlike the objfunction() method which accepts one point at a time to be evaluated (if I understand the ReadTheDocs documentation properly). Also, it seems to me that it is possible that objfunction() and eval_ineq_constraints() are not necessarily called as a pair - objfunction(x) could be called multiple times for different x before a single call to eval_ineq_constraints() might be performed which evaluates the constraint functions for multiple points during that single call to eval_ineq_constraints().

I think it is possible for a user to write their objfunction() and eval_ineq_constraints() in a clever way to be able to make efficient use of each call to the external simulation program, e.g. storing both objective and constraint function values from an external simulation run for a point even, if invoked by only one of objfunction() or eval_ineq_constraints(); hence the stored values can be checked by future objfunction() or eval_ineq_constraints() calls later in the optimisation to avoid repeating external simulations if we already have the information stored by an earlier call.

In addition, I have another related question: if there are to be 500 objective function evaluations in an optimisation, then there would also be 500 evaluations for each constraint, where the objective and constraint functions are evaluated on the exact same set of points? I am not familiar enough with either optimisation in general or pySOT in particular to know for sure.

Many thanks.

CandidateDYCORS implements which strategy?

In the documentation it says:

CandidateDYCORS Uses a DDS strategy...

however after reading the code and the DYCORS paper (Regis & Shoemaker, 2012), the search strategy it's actually DYCORS-LMSRBF, not DYCORS-DDSRBF.

Please confirm.

Thanks