During AugLagrangian's optimization, L_BFGS sets coordinates to nan about ensmallen

[olgavrou]

OK will do :) I think the first solution is probably working better, I got another nan with the second one at some point

[conradsnicta]

PS. While we are at it, the following line should be replaced:
https://github.com/mlpack/ensmallen/blob/master/include/ensmallen_bits/lbfgs/lbfgs_impl.hpp#LL93C1-L93C57

 scalingFactor = 1.0 / sqrt(dot(gradient, gradient));

with

 scalingFactor = 1.0 / arma::norm(gradient, "fro");

This is for two reasons: (1) clarity of intent, (2) Armadillo will use a more robust algorithm to calculate the norm.

[conradsnicta]

Should be resolved in #368. If not, please re-open and provide more details.

[rcurtin]

Thanks for the clear report! I agree that this is a problem and I like your suggested fixes. I might pick the first solution as it will still compute a nonzero scaling factor if || yMat || is very small, but I don't have a particularly strong opinion and could be convinced either way. If you'd like to open a PR I would gladly review it and we can get the fix merged. Thanks again! 👍

[conradsnicta]

scalingFactor = dot(sMat, yMat) / std::max(1e-12, dot(yMat, yMat));

Instead of a hardcoded value like 1e-12, suggest to use someting like
1000 * std::numeric_limits<CubeElemType>::epsilon().

The user may elect to use matrices and cubes with single-precision floating point values, rather than double-precision. In other words, CubeElemType can be either float or double. For single-precision, the hardcoded 1e-12 is probably too low in this context.

[mlpack-bot]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions! 👍