Optimization is very slow about bayesianoptimization

[jtromans]

I've seen this behavior with other implementations too. My understanding is that the time taken to calculate the GP will scale cubically with the number of observations. However, the actual calculation itself should be relatively quick, especially compared to the black-box function.

[fmfn]

I resisted intervening here since the optimizer could be made faster by the user. However I decided to bite the bullet and make the changes. See #176

[KOLANICH]

@KOLANICH what other optimizers have you used? any of them do Bayesian optimization?

Here is the list: https://gitlab.com/KOLANICH/UniOpt.py/tree/master/UniOpt/backends

[fmfn]

You control the training of the underlying Gaussian Process by passing **gp_params to the maximize method. The default behaviour included 25 rounds of pseudo-likelihood optimization, which happening internally within sklearn's GP.

Is this pattern consistent? GPs are finicky, sometime I find you get odd numerical instabilities throwing you off. Increasing alpha usually helps, since it makes the process of inverting the covariance matrix more stable.

[tachim]

I'm also seeing this behavior, and increasing alpha doesn't do anything

[cotterpl]

I'm also seeing this behavior. Alpha doesn't change anything:

from bayes_opt import BayesianOptimization

def toOptimize(a=0):
    return - a * a

optimizer = BayesianOptimization(toOptimize,{'a': [-5,5]})
optimizer.maximize(n_iter=200)
print(optimizer.res)

Initialization
-----------------------------------------
 Step |   Time |      Value |         a | 
    1 | 00m00s |  -22.89916 |   -4.7853 | 
    2 | 00m00s |   -1.10813 |    1.0527 | 
    3 | 00m00s |   -0.01665 |    0.1290 | 
    4 | 00m00s |   -4.29761 |    2.0731 | 
    5 | 00m00s |   -0.07172 |    0.2678 | 
Bayesian Optimization
-----------------------------------------
 Step |   Time |      Value |         a | 
    6 | 00m00s |  -25.00000 |    5.0000 | 
    7 | 00m00s |   -0.49688 |   -0.7049 | 
    8 | 00m01s |   -0.03670 |   -0.1916 | 
    9 | 00m02s |   -0.00059 |   -0.0243 | 
   10 | 00m03s |   -0.00044 |    0.0210 | 

And it keeps getting slower with each iteration

[jjiteshh]

me too! it gets too slow after a while..

[hofesh]

Seeing same behaviour. Takes several seconds for each iteration even with a dummy target function.

[KOLANICH]

In fact it is the slowest optimizer of all I have ever used. And the results are worse than the ones of the faster optimizers on the same iteration count. So I personally have never used it in production.

[nshervt]

In fact it is the slowest optimizer of all I have ever used. And the results are worse than the ones of the faster optimiz3rs on the same iteration count. So I personally have never used it in production.

@KOLANICH what other optimiz3rs have you used? any of them do Bayesian optimization?

[louis925]

I also see the same slow down behavior at around 400+ iterations with 10 parameters to being optimized. Now the 5 fold training of my model take much less time than the calculation of Bayesian optimization.