Failure to converge (again...) about lbfgspp

[yixuan]

It might be numerical issues. The Gaussian kernel computes the exponential function of the squared distance. For example, in your example the distance between -4 and 4 is 8, so when l=1, the kernel computes exp(-8^2/2)=1.266417e-14. Then when you invert the kernel matrix you get pretty large gradient, and the line search may fail. I agree that some more mature software may handle this better, but given the nature of the kernel function, there can be lots of instabilities. A good initial value is always appreciated.

Specific to your problem, let me think of a better way to computing the gradient.

[Svalorzen]

I think you might be right. Now I realize that the Python code actually uses numerical gradients (as I don't pass it the derivative), so it might be able to avoid the numerical instabilities at the price of performance?

I am looking at GPy's code also to see if I can find a different way to compute the gradient..