mlpack / ensmallen Goto Github PK
View Code? Open in Web Editor NEWA header-only C++ library for numerical optimization --
Home Page: http://ensmallen.org
License: Other
A header-only C++ library for numerical optimization --
Home Page: http://ensmallen.org
License: Other
The Debian ensmallen package is automatically built (compiled) on many architectures. The build process runs the test scripts. Everything works fine on some architectures (amd64, say) albeit with a rather large amount of time required for the test phase. But on some other architectures (i386, see https://buildd.debian.org/status/fetch.php?pkg=ensmallen&arch=i386&ver=2.10.0-1&stamp=1568047415 for details) the test phase exceeds some resource limitation and the build is terminated.
I could just turn off testing, but that seems wrong. Best would be if the test phase were much faster, or if there was a fast test run by default with the current extremely time consuming test run only if some extra option is given.
Clone https://github.com/barak/ensmallen, go to tag debian/2.10.0-1, run fakeroot debian/rules binary
on a crappy machine with a low ulimit set.
ensmallen sucessfully autobuilt on all sorts of weird architectures.
Hi,
After having solved an optimization problem, is it a way to estimate the gradient and the hessian matrix at the optimum point it the function does not implement the gradient nor the hessian?
Best Regards
There are several header files with using namespace std;
. This is a bad practise because it pollutes the application namespace when including those header files in your application.
Here is the list of the header files with such a definition:
ensmallen_bits/problems/ackley_function_impl.hpp
ensmallen_bits/problems/beale_function_impl.hpp
ensmallen_bits/problems/cross_in_tray_function_impl.hpp
ensmallen_bits/problems/goldstein_price_function_impl.hpp
ensmallen_bits/problems/himmelblau_function_impl.hpp
ensmallen_bits/problems/holder_table_function_impl.hpp
ensmallen_bits/problems/levy_function_n13_impl.hpp
ensmallen_bits/problems/schaffer_function_n2_impl.hpp
ensmallen_bits/problems/schaffer_function_n4_impl.hpp
ensmallen_bits/problems/three_hump_camel_function_impl.hpp
include ensmallen header file in the code
Including ensmallen in your code must not make std namespace to be used by default.
Including ensmallen in your code causes std namespace to be used by default.
Ensmallen currently lacks optimizers which are able to find multiple optima to multimodal functions. We could find multiple optima through niching or by having an implementation of Glowworm Swarm Optimization. Please let me know your thoughts on this.
ensmallen/include/ensmallen_bits/ens_version.hpp
Lines 15 to 17 in 51dbf63
Even if somebody processes the ENS_VERSION_PATCH macro (instead of ens::version::patch), the patch version number should still be an (unsigned) integer for consistency.
I suggest marking a release as a RC within the ENS_VERSION_NAME macro:
For example, version 2.399.1 could be marked as RC1 for a new upcoming version 2.400.0.
For example:
cout << ens::version::as_string() << endl;
output: 2.399.1 (2.400-RC1)
Having a fixed 3 digit number (>=100) for the minor version number offers a lot of flexibility, all while staying consistent with semantic versioning :)
(related to #27)
This is ported from mlpack/mlpack#395. There is some more discussion there. Here is the original description from @stephentu:
SGD for LR-SDP is becoming more popular, and this paper http://arxiv.org/pdf/1411.1134v2.pdf lays the groundwork for some neat theoretical convergence results (in particular, how to choose step sizes). This is really cool stuff which we should investigate.
This could be a fun project for someone looking to contribute code to mlpack.
The ensmallen documentation is full of big example programs like the one at the bottom of this section:
http://ensmallen.org/docs.html#differentiable-functions
These examples are kind of hard to scroll through when perusing the documentation. It would be really cool if we had a way to make these collapsible. The answer might actually be in the ensmallen.org repo, but I am not sure if Markdown has support for collapsible code snippets or anything. In any case, I haven't looked into it and don't have time to do it now, but thought it would be a really nice feature so I opened it as an issue in case anyone is interested. If not I will get around to it eventually. :-D
Probably it's not necessary to have everything be collapsible, only the "long" example programs.
Each optimizer currently has a signature like
template<typename FunctionType>
void Optimize(FunctionType& f, arma::mat& coordinates);
but really it might be nicer if we did
template<typename FunctionType, typename MatType>
void Optimize(FunctionType& f, MatType& coordinates);
The code change should be straightforward for each optimizer but there is one main issue from my end: all the documentation as written suggests writing FunctionType
functions with arma::mat
. Making that documentation fully templatized would make it harder to provide really simple examples, so I have to wonder if adding some extra section on "advanced functionality" that mentions that different types of Armadillo matrices can typically be used for optimization. If anyone has any other ideas, feel free to propose them.
Also, it's probably worth writing at least one test for a different-precision Armadillo matrix for each optimizer, but existing tests can be easily adapted, I think. (I suppose also that means that the Evaluate()
and Gradient()
for everything in problems/
will need to be adapted, but again that too should be straightforward.)
This is a continued version of mlpack/mlpack#1475, ported to the ensmallen repository because this is a more appropriate place. This was originally opened by @FloopCZ with the following text:
Hi everyone, first of all, thank you for the huge amount of work you have done on this project.
After a brief pass through the CMA-ES implementation, I found a few inconsistencies with the documentation.
The documentation of the Optimize function states that the last parameter (iterate) is also used as the starting point of the algorithm. However, this parameter seems to be basically ignored and the starting point is generated randomly here.
When the algorithm does not converge, it suggests to use a smaller step size here, yet there is no way to change the step size.
The lowerBound and upperBound variables don't seem to be true bounds for the variables. As far as I understand the code, those are only used to deduce the initial sigma value and the random starting point.
Have a nice day!
There are some responses in the original issue about how we should handle each point, but the issue is not yet solved.
Many of the tests rely on functions in the LogisticRegression class that are currently not present in ensmallen. For example, ada_delta_test.cpp
uses LogisticRegression::ComputeAccuracy()
. I think it's necessary to port these from mlpack in order to keep the tests working. (It would also be quicker than rewriting the tests).
In the same vein, many tests also rely on GaussianDistribution
, which is also currently not present in ensmallen. I had a look at the implementation in mlpack, and it shouldn't be too difficult to port it over to ensmallen. A few parts need to be rewritten, to avoid calling into other innards of mlpack. For example, mlpack/src/mlpack/core/dists/gaussian_distribution.cpp
calls gmm::PositiveDefiniteConstraint::ApplyConstraint()
.
Stuff like GaussianDistribution::serialize()
can presumably be simply stripped out.
@zoq - Since you're more familiar with the mlpack code base than me, could you port GaussianDistribution and/or the necessary functions to LogisticRegression?
At the end of most of the SGD-like optimizers, we have a loop that loops over all parts of the decomposable function and computes the final objective. But this is problematic in that the decomposable function could have very many functions, meaning that the final objective computation takes a very long time. This could easily be seen with, e.g., training a machine learning model, where you might take 20 passes over the data and each pass could take an hour or more. Computing the exact final objective is equivalent to one additional pass over the data.
To solve this, we could add an extra parameter to each of our decomposable function optimizers: exactObjective
. If exactObjective
is true, then the final objective will be computed exactly, just like it is now. If it's false (which should be the default), then just the estimate of the final objective obtained on the last pass over the data will be used.
In general, only estimates of the final objective function are given back by all deep learning toolkits and machine learning optimization toolkits I have encountered. So computing the exact objective every time is basically just a computation disadvantage for us, and although it can be useful, we should allow the user to specify whether they want it (and assume that they don't, to match what others are doing).
I installed "ensmallen" successfully. Then an error occur when i calling by "using namespace ens".
The error is following:
Error C2664 'arma::Mat::Mat(arma::Mat &&)': cannot convert argument 1 from 'arma::uword' to 'const arma::SizeMat &'
at line 92 in file "atoms.hpp" :
arma::mat gradient(size(x));
When using callbacks like ens::ProgessBar
or similar, and the optimization terminates partway through an epoch due to a convergence check, the output can be confusing:
Epoch 1/29[=========================================================================================>..........] 89% - ETA: 0s - loss: 1115.75
but then output stops.
See the discussion at mlpack/mlpack#2073 for many more details and how to reproduce.
We should see if we can modify the ens::ProgressBar
callback (and perhaps others) to give better output in these situations. Perhaps something like this would be an improvement:
Epoch 1/29[=========================================================================================>..........] 89% - ETA: 0s - loss: 1115.75
Optimization converged! (value 1.33e-6, tolerance 1e-5)
Instead the output just terminates:
Epoch 1/29[=========================================================================================>..........] 89% - ETA: 0s - loss: 1115.75
This would be a good issue if you are not familiar with how ensmallen's callbacks work and would like to be. I don't have a direct route to a solution here, so my suggestion would be to investigate the current callback code, reproduce the issue, then think about the cleanest way to print convergence information in the ens::ProgressBar
callback. Once that's done, see if similar changes might be useful for some of the other callbacks that print information (see http://ensmallen.org/docs.html#callback-documentation for more information on the callbacks that are available).
Let's see if mlpack-bot labels it.
The paper sources don't serve a directly useful purpose to the users of the software. Having a single pdf would be more useful (ie. it would be immediately viewable).
It would be good to clean this up and move the sources to a separate repo, in similar vein to the joss_paper for mlpack. As an added bonus, the modified arxiv version of the paper can be kept in the same repo.
(PS. I'm happy to do the actual moving / cleanup / etc.)
I don't have any problems with version numbers like Armadillo and I can see why this is useful, but I realized while browsing http://semver.org today that this actually violates their guidelines:
2. A normal version number MUST take the form X.Y.Z where X, Y, and Z are non-negative
integers, and MUST NOT contain leading zeroes. X is the major version, Y is the minor
version, and Z is the patch version. Each element MUST increase numerically. For
instance: 1.9.0 -> 1.10.0 -> 1.11.0.
In this case, I think it might be more important to go with the standard instead of needing to justify why we're doing "semantic versioning except this one bit". Lots of tools that deal with versions have specific version functions for greater than, less than, etc., such as CMake which has VERSION_GREATER
as a comparison operator. Thoughts? I would say my mild preference is to stick with the original semantic versioning scheme as written, just to avoid any confusion or need for extra documentation.
The PrimalDual
solver in ensmallen_bits/sdp/
uses a lot of functionality out of mlpack's linear algebra support; specifically, this can be seen here:
$ grep 'math::' ensmallen_bits/sdp/*
primal_dual_impl.hpp: math::Smat(F * rd - rc, Frd_rc_Mat);
primal_dual_impl.hpp: math::Svec(Einv_Frd_rc_Mat, Einv_Frd_rc);
primal_dual_impl.hpp: math::Smat(F * (rd - Asparse.t() * dysparse - Adense.t() * dydense) - rc,
primal_dual_impl.hpp: math::Svec(Einv_Frd_ATdy_rc_Mat, Einv_Frd_ATdy_rc);
primal_dual_impl.hpp: math::Svec(sdp.SparseA()[i], Aisparse);
primal_dual_impl.hpp: math::Svec(sdp.DenseA()[i], Aidense);
primal_dual_impl.hpp: math::Svec(sdp.C(), sc);
primal_dual_impl.hpp: math::Svec(X, sx);
primal_dual_impl.hpp: math::Svec(Z, sz);
primal_dual_impl.hpp: math::SymKronId(X, F);
primal_dual_impl.hpp: math::Svec(Gk, gk);
primal_dual_impl.hpp: math::Svec(Gk, gk);
primal_dual_impl.hpp: math::Svec(Rc, rc);
primal_dual_impl.hpp: math::Smat(dsx, dX);
primal_dual_impl.hpp: math::Smat(dsz, dZ);
primal_dual_impl.hpp: math::Svec(Rc, rc);
primal_dual_impl.hpp: math::Smat(dsx, dX);
primal_dual_impl.hpp: math::Smat(dsz, dZ);
primal_dual_impl.hpp: math::Svec(X, sx);
primal_dual_impl.hpp: math::Svec(Z, sz);
sdp_impl.hpp: math::Svec(arma::mat(SparseA()[i]), sa);
sdp_impl.hpp: math::Svec(DenseA()[i], sa);
This should be pretty straightforward, we just need to import the implementations of these functions into a header file in this directory and change the call or something.
There are a couple instances where mlpack's Random()
or RandInt()
functions are used:
./parallel_sgd/parallel_sgd_impl.hpp: mlpack::math::randGen);
./sa/sa_impl.hpp: const double unif = 2.0 * math::Random() - 1.0;
./sa/sa_impl.hpp: const double xi = math::Random();
./scd/descent_policies/random_descent.hpp: return mlpack::math::RandInt(function.NumFeatures());
I think this shows up a lot in the tests too. Instead, we can use Armadillo's RNG here; so mlpack::math::randGen
can (probably) be replaced by arma::arma_rng_cxx11_instance
or something similar, and we can use that to generate random numbers.
However there is also the Armadillo configuration parameter ARMA_USE_EXTERN_CXX11_RNG
; but, looking through it quickly, I don't see anything different for us to do whether or not someone using ensmallen has written #define ARMA_USE_EXTERN_CXX11_RNG
.
The (temporary) tests_catch/
directory contains the converted tests from boost to catch. This is currently a work in progress.
I've converted about a third of the tests and came up with the following conversion rules. I'm not sure whether these are completely correct, so it would be useful to check these.
BOOST_AUTO_TEST_SUITE(blah)
-> delete, but "blah" is used as the second argument to all instances of TEST_CASE()
within a given file
BOOST_AUTO_TEST_SUITE_END()
-> delete
BOOST_AUTO_TEST_CASE(testname)
-> TEST_CASE("testname", "[blah]")
following the examples given in the Catch2 tutorial
BOOST_REQUIRE_SMALL(x, 0.015)
-> REQUIRE(x == Approx(0.0).margin(0.015))
as per the docs for Catch2 assertion macros; not sure whether this is the best or even the correct approach
BOOST_REQUIRE_CLOSE(x, 1.23, 0.15)
-> REQUIRE(result == Approx(1.23).epsilon(0.0015))
Boost uses the "0.15" to express 0.15%. I'm not certain on the Catch2 equivalent expression. I've followed the example in the docs for assertion macros, which uses .epsilon(0.01)
to express a 1% difference. The Catch2 docs appear unclear on the exact meaning of .epsilon(val)
: "epsilon serves to set the coefficient by which a result can differ from Approx's value before it is rejected." Coefficient of what?
BOOST_REQUIRE_EQUAL(i, 123)
-> REQUIRE(i == 123)
BOOST_REQUIRE_GT(x, 0.05)
-> REQUIRE(x > 0.05)
BOOST_REQUIRE_GE(x, 0.05)
-> REQUIRE(x >= 0.05)
BOOST_REQUIRE_LT(x, 0.05)
-> REQUIRE(x < 0.05)
BOOST_REQUIRE_LE(x, 0.05)
-> REQUIRE(x <= 0.05)
I think right now we can keep the documentation relatively simple, I think something like this could work?
Adadelta is an extension of Adagrad that adapts learning rates based on a moving window of gradient updates, instead of accumulating all past gradients. Instead of accumulating all past squared gradients, the sum of gradients is recursively defined as a decaying average of all past squared gradients.
AdaDelta(stepSize, batchSize, rho, epsilon , maxIterations, tolerance, shuffle);
AdaDelta(stepSize, batchSize);
AdaDelta(stepSize);
**stepSize** (double) | Step size for each iteration. |
**batchSize** (int) | Number of points to process in one step. |
**rho** (double) | Smoothing constant. Corresponding to fraction of gradient to keep at each time step. |
**epsilon** (double) | Value used to initialise the mean squared gradient parameter. |
**maxIterations** (int) | Maximum number of iterations allowed (0 means no limit). |
**tolerance** (double) | Maximum absolute tolerance to terminate algorithm. |
**shuffle** (bool) | If true, the function order is shuffled; otherwise, each function is visited in linear order. |
AdaDelta optimizer(1.0, 1, 0.99, 1e-8, 5000000, 1e-9, true)
RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();
optimizer.Optimize(f, coordinates);
Start 1: ensmallen_tests
1/1 Test #1: ensmallen_tests ..................***Failed 363.56 sec
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 363.57 sec
The following tests FAILED:
1 - ensmallen_tests (Failed)
Errors while running CTest
OS: FreeBSD
So I have been looking for examples on how to use the different methods of FFN wrapper other than Add and Train no luck so far. So, I decided to give it a go on my understanding and so far it is going horrible.
I planned on implementing ResNet
now all I wanted to do was
output=convolution(prev_input+prev_output)
but all the examples are straight forward model.Add and then optimize
So I planned to first add all the conv layer to the model by model.Add< xyzlayer<> >()
Then forward ops in a for loop followed by a loss function and then optimizer.optimize
So here is a what I wrote:
//these are methods of ResNet Struct
mat conv_forward(mat input){
mat output;
//We are trying to go just one convolution ahead;
model.Forward(input,output,iter,iter);
iter++;
return output
}
void train(mat input, mat target){
mat params =model.Parameters();
error=this->loss_func(input, target);
optimizer.optimize(Params,error);
cout<<"iter :\n";
}
double loss_func(mat input, mat target){
loss_val=loss.Forward(input,target);
return loss_val;
}
A little Insight would be very helpfull.
ThankYou.
How were the visualizations done for this interactive demo?
Is the source available?
Hi everyone, I was training an LSTM using mlpack. A portion of the code is given below:
optimizer.Tolerance() = -1;
optimizer.MaxIterations() = 0;
model.Train(trainX,
trainY,
optimizer,
ens::PrintLoss(),
ens::ProgressBar(),
ens::EarlyStopAtMinLoss(100),
ens::StoreBestCoordinates<arma::mat>());
The model converges at a local minima if patience is very but overfits with a patience of 100. I was wondering if there was a way to terminate it after say 100th epoch irrespective of whether it reached a minima.
This way we could avoid using for loops
altogether.
I was trying to find a solution such that I could use EarlyStopAtMinLoss and not exceed 100 epochs and not use for loops
.
One solution would be to add a condition in the EndEpoch.
I am not sure if this necessary, So I would love to hear your opinion.
If a way of doing the same doesn't exist, can I implement the same.
Sorry if I missed something.
Thanks.
The examples on the webpage use #include <ensmallen>
instead of #include <ensmallen.hpp>
, but we only provide ensmallen.hpp
. So I think we should either adjust the examples or change the name of the header file. I don't have any preference, any thoughts?
When pulling from HISTORY.md
, the release notes:
ensmallen.hpp
and GitHub release page.#86
-> [#86](https://github.com/mlpack/ensmallen/pull/86)
Would it be possible to always include these components?
On including ensmallen.hpp, the included header files are unable to "find" the header files that they include, since the file paths seem wrong. This is fixed by using relative paths instead.
Sample code:
`#include "ensmallen.hpp"
int main()
{
RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();
CNE optimizer(200, 10000, 0.2, 0.2, 0.3, 65, 0.1e-4);
optimizer.Optimize(f, coordinates);
return 0;
}`
Error in terminal:
In file included from ensmallen.hpp:64:0,
from test.cpp:1:
ensmallen_bits/ada_delta/ada_delta.hpp:19:38: fatal error: ensmallen_bits/sgd/sgd.hpp: No such file or directory
compilation terminated.
I am getting linking error while calling Optimize function for any ensmallen optimizer.
I coded as stated in http://ensmallen.org/docs.html, infact I copy pasted the same code but I get linking error while calling the Optimize function.
I am using Windows 10 and Visual studio 2017
I got these linking errors in the SGD.obj file (here SGD is my program name) ->
1>SGD.cpp
1>SGD.obj : error LNK2019: unresolved external symbol sasum_ referenced in function "double __cdecl arma::blas::asum(unsigned __int64,double const *)" (??$asum@N@blas@arma@@YAN_KPEBN@Z)
1>SGD.obj : error LNK2019: unresolved external symbol dasum_ referenced in function "double __cdecl arma::blas::asum(unsigned __int64,double const *)" (??$asum@N@blas@arma@@YAN_KPEBN@Z)
1>SGD.obj : error LNK2019: unresolved external symbol snrm2_ referenced in function "double __cdecl arma::blas::nrm2(unsigned __int64,double const *)" (??$nrm2@N@blas@arma@@YAN_KPEBN@Z)
1>SGD.obj : error LNK2019: unresolved external symbol dnrm2_ referenced in function "double __cdecl arma::blas::nrm2(unsigned __int64,double const *)" (??$nrm2@N@blas@arma@@YAN_KPEBN@Z)
1>SGD.obj : error LNK2019: unresolved external symbol sdot_ referenced in function "double __cdecl arma::blas::dot(unsigned __int64,double const *,double const *)" (??$dot@N@blas@arma@@YAN_KPEBN1@Z)
1>SGD.obj : error LNK2019: unresolved external symbol ddot_ referenced in function "double __cdecl arma::blas::dot(unsigned __int64,double const *,double const *)" (??$dot@N@blas@arma@@YAN_KPEBN1@Z)
1>SGD.obj : error LNK2019: unresolved external symbol sgemv_ referenced in function "void __cdecl arma::blas::gemv(char const *,int const *,int const *,double const *,double const *,int const *,double const *,int const *,double const *,double *,int const *)" (??$gemv@N@blas@arma@@YAXPEBDPEBH1PEBN21212PEAN1@Z)
1>SGD.obj : error LNK2019: unresolved external symbol dgemv_ referenced in function "void __cdecl arma::blas::gemv(char const *,int const *,int const *,double const *,double const *,int const *,double const *,int const *,double const *,double *,int const *)" (??$gemv@N@blas@arma@@YAXPEBDPEBH1PEBN21212PEAN1@Z)
1>SGD.obj : error LNK2019: unresolved external symbol cgemv_ referenced in function "void __cdecl arma::blas::gemv(char const *,int const *,int const *,double const *,double const *,int const *,double const *,int const *,double const *,double *,int const *)" (??$gemv@N@blas@arma@@YAXPEBDPEBH1PEBN21212PEAN1@Z)
1>SGD.obj : error LNK2019: unresolved external symbol zgemv_ referenced in function "void __cdecl arma::blas::gemv(char const *,int const *,int const *,double const *,double const *,int const *,double const *,int const *,double const *,double *,int const *)" (??$gemv@N@blas@arma@@YAXPEBDPEBH1PEBN21212PEAN1@Z)
1>SGD.obj : error LNK2019: unresolved external symbol sgemm_ referenced in function "void __cdecl arma::blas::gemm(char const *,char const *,int const *,int const *,int const *,double const *,double const *,int const *,double const *,int const *,double const *,double *,int const *)" (??$gemm@N@blas@arma@@YAXPEBD0PEBH11PEBN21212PEAN1@Z)
1>SGD.obj : error LNK2019: unresolved external symbol dgemm_ referenced in function "void __cdecl arma::blas::gemm(char const *,char const *,int const *,int const *,int const *,double const *,double const *,int const *,double const *,int const *,double const *,double *,int const *)" (??$gemm@N@blas@arma@@YAXPEBD0PEBH11PEBN21212PEAN1@Z)
1>SGD.obj : error LNK2019: unresolved external symbol cgemm_ referenced in function "void __cdecl arma::blas::gemm(char const *,char const *,int const *,int const *,int const *,double const *,double const *,int const *,double const *,int const *,double const *,double *,int const *)" (??$gemm@N@blas@arma@@YAXPEBD0PEBH11PEBN21212PEAN1@Z)
1>SGD.obj : error LNK2019: unresolved external symbol zgemm_ referenced in function "void __cdecl arma::blas::gemm(char const *,char const *,int const *,int const *,int const *,double const *,double const *,int const *,double const *,int const *,double const *,double *,int const *)" (??$gemm@N@blas@arma@@YAXPEBD0PEBH11PEBN21212PEAN1@Z)
1>SGD.obj : error LNK2019: unresolved external symbol ssyrk_ referenced in function "void __cdecl arma::blas::syrk(char const *,char const *,int const *,int const *,double const *,double const *,int const *,double const *,double *,int const *)" (??$syrk@N@blas@arma@@YAXPEBD0PEBH1PEBN212PEAN1@Z)
1>SGD.obj : error LNK2019: unresolved external symbol dsyrk_ referenced in function "void __cdecl arma::blas::syrk(char const *,char const *,int const *,int const *,double const *,double const *,int const *,double const *,double *,int const *)" (??$syrk@N@blas@arma@@YAXPEBD0PEBH1PEBN212PEAN1@Z)
1>SGD.obj : error LNK2019: unresolved external symbol sgesdd_ referenced in function "void __cdecl arma::lapack::gesdd(char *,int *,int *,double *,int *,double *,double *,int *,double *,int *,double *,int *,int *,int *)" (??$gesdd@N@lapack@arma@@YAXPEADPEAH1PEAN1221212111@Z)
1>SGD.obj : error LNK2019: unresolved external symbol dgesdd_ referenced in function "void __cdecl arma::lapack::gesdd(char *,int *,int *,double *,int *,double *,double *,int *,double *,int *,double *,int *,int *,int *)" (??$gesdd@N@lapack@arma@@YAXPEADPEAH1PEAN1221212111@Z)
1>SGD.obj : error LNK2019: unresolved external symbol sgesv_ referenced in function "void __cdecl arma::lapack::gesv(int *,int *,double *,int *,int *,double *,int *,int *)" (??$gesv@N@lapack@arma@@YAXPEAH0PEAN00100@Z)
1>SGD.obj : error LNK2019: unresolved external symbol dgesv_ referenced in function "void __cdecl arma::lapack::gesv(int *,int *,double *,int *,int *,double *,int *,int *)" (??$gesv@N@lapack@arma@@YAXPEAH0PEAN00100@Z)
1>SGD.obj : error LNK2019: unresolved external symbol cgesv_ referenced in function "void __cdecl arma::lapack::gesv(int *,int *,double *,int *,int *,double *,int *,int *)" (??$gesv@N@lapack@arma@@YAXPEAH0PEAN00100@Z)
1>SGD.obj : error LNK2019: unresolved external symbol zgesv_ referenced in function "void __cdecl arma::lapack::gesv(int *,int *,double *,int *,int *,double *,int *,int *)" (??$gesv@N@lapack@arma@@YAXPEAH0PEAN00100@Z)
1>SGD.obj : error LNK2019: unresolved external symbol sposv_ referenced in function "void __cdecl arma::lapack::posv(char *,int *,int *,double *,int *,double *,int *,int *)" (??$posv@N@lapack@arma@@YAXPEADPEAH1PEAN1211@Z)
1>SGD.obj : error LNK2019: unresolved external symbol dposv_ referenced in function "void __cdecl arma::lapack::posv(char *,int *,int *,double *,int *,double *,int *,int *)" (??$posv@N@lapack@arma@@YAXPEADPEAH1PEAN1211@Z)
1>SGD.obj : error LNK2019: unresolved external symbol cposv_ referenced in function "void __cdecl arma::lapack::posv(char *,int *,int *,double *,int *,double *,int *,int *)" (??$posv@N@lapack@arma@@YAXPEADPEAH1PEAN1211@Z)
1>SGD.obj : error LNK2019: unresolved external symbol zposv_ referenced in function "void __cdecl arma::lapack::posv(char *,int *,int *,double *,int *,double *,int *,int *)" (??$posv@N@lapack@arma@@YAXPEADPEAH1PEAN1211@Z)
1>C:\Users\Gaurav Sharma\Documents\Visual Studio 2017\Projects\ensmallenProjects\SGD\x64\Debug\SGD.exe : fatal error LNK1120: 26 unresolved externals
1>Done building project "SGD.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========
Hello,
Thanks for releasing ensmallen 2.10.0 with Callback PR, it is really a great work.
It is possible to show an example about how to use the callbacks with ann? I am not sure if I understood well the ffn class, but It seems that the train function does not take in account the callbacks yet, since it is calling optimizer.Optimize(*this, parameter)
Best regards,
This is a port of mlpack/mlpack#401. There are some more details there. This is the original description from @stephentu:
Our SDP abstraction right now only supports sparse and dense matrices. This is sufficiently general, but for some certain classes of constraints, quite suboptimal.
As an example, suppose I want to have the constraint that the sum of all elements in the matrix is constant. While we can express this as Tr((11^T) * X) = b, this is quite inefficient to evaluate for something that could be expressed as X.sum().
Hence, it would be desirable to have a third type of constraint, "linearOperator", which is just a linear function which takes symmetric matrices to real numbers. The straightforward way to implement this would be a vector of std::function objects.
Many of the original mlpack tests declare the use of a lot of namespaces. For example:
using namespace mlpack::optimization;
using namespace mlpack::optimization::test;
using namespace mlpack::distribution;
using namespace mlpack::regression;
using namespace mlpack::ann;
using namespace mlpack::cv;
This raises a few questions:
ens
), or are nested namespaces going to be kept? I think the former would keep things simpler for end users, and it wouldn't require much code change -- in ensmallen.hpp
and within the ens
namespace, simply do a few using namespace ens::blah
linesmlpack::distribution
and mlpack::regression
going to be kept?tests
directory required? For example, going by the using namespace
declarations in cv_test.cpp
, it seems that a good chunk of mlpack is being used and/or tested.To maximise impact, I suggest expanding the github repo description from:
A header-only C++ library for mathematical optimization.
to
A header-only C++ library for mathematical optimization -- http://ensmallen.org
In the Constrained functions documentation it specifies this function prototype:
double GradientConstraint(const size_t i, const arma::mat& x, arma::mat& g);
As per the source code this should be:
void GradientConstraint(const size_t i, const arma::mat& x, arma::mat& g);
Since we aren't monitoring memory now, we recently had a memory leak (#167). It's probably a good idea to copy the mlpack valgrind ci test to the ensmallen repo too.
For the bounded problem, the L-BFGS-B method is a good option, which has not been included in the library yet. while a Fortran version is out there (Fortran code for L-BFGS-B), is there any plan to implement the L-BFGS-B method in this library?
At least one of the tests uses data::Load()
from mlpack:
data::Load("sensing_X.csv", Xorig, true, false);
We can replace this with direct armadillo CSV loading + transposes etc. Example:
arma::mat Xorig;
Xorig.load("sensing_X.csv", arma::csv_ascii);
Xorig = Xorig.t();
...
Any thoughts?
Related to issue #3.
If you run CMAES with different seeds every time with the code:
`#include "ensmallen.hpp"
#include
int main()
{
ens::test::RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();
arma::arma_rng::set_seed_random();
ens::CMAES<> optimizer(0, -10, 10, 32, 200, -1);
double value = optimizer.Optimize(f, coordinates);
coordinates.print();
std::cout << value;
}
`
and then executing, you sometimes get the error:
error: chol(): decomposition failed
terminate called after throwing an instance of 'std::runtime_error'
what(): chol(): decomposition failed
Aborted (core dumped)
signifying that the cholesky decomposition in line 131 of cmaes_impl.cpp has failed.
Pretty much all the sources in ensmallen_bits
have #include <armadillo>
.
It would probably make sense to simplify the code by doing the inclusion only once, at the top of ensmallen.hpp
.
In a couple of files we use mlpack's logging capabilities (Log::Info, Log::Warn
). Are we going to use: arma::get_cout_stream()
and arma::get_cerr_stream()
. Any thoughts?
Hi,
I was testint how the library works on Windows with Microsoft Visual Studio 2017. I compile every library manually: armadillo 9.8 and openblas 0.3.7 according to the documentation.
However, when i build a simple example:
#include <ensmallen.hpp>
#include <armadillo>
class SquaredFunction
{
public:
double Evaluate(const arma::mat& x)
{
return 2 * std::pow(arma::norm(x), 2.0);
}
};
int main()
{
arma::mat x("1.0 -1.0 1.0");
ens::CNE optimizer;
SquaredFunction f; // Create function to be optimized.
optimizer.Optimize(f, x);
std::cout << "Minimum of squared function found with simulated annealing is " << x;
return 0;
}
Moreover, I have linked armadillo.lib
, openblas.lib
and lapack.lib
. Unfourtunately, I get two linker error:
Error LNK2001 unresolved external symbol sgesdd_
Error LNK2001 unresolved external symbol dgesdd_
I imagine that I am not building something properly, but I have tried to everything.
When we run SGD with 0 batch size then the loop goes infinite. Although it is very stupid thing to use 0 batch size but user can accidentally set batch size to 0 ( as I did it accidentally ). And as training on very large data set and complex machine learning models do take large amount of time so if an user accidentally set batch size 0 for such situation then they may think that the optimizer is taking so long time probably due to large data set and complex model but indeed they were stuck in a infinite loop and were not aware.
And if we set batch size to 0 then logically our optimizer shouldn't do anything i.e., it will not update the iterate ( initial points given by user ) and the overallObjective return by the optimizer is the value of the function which is to be optimized at that initial point given by the user.
Apart from this as software engineering tells us that while testing we should handle such boundary conditions.
The shuffle
parameter in spsa
is not used.
../inst/include/ensmallen_bits/spsa/spsa.hpp:120:8:
warning: private field 'shuffle' is not used [-Wunused-private-field]
bool shuffle;
^
Compile ensmallen.
The differential evolution algorithm is documented as needing an arbitrary function, both in the function type and optimizer section of the documentation.
However, using the code in the master branch (commit 9b730c7), the DE algorithm need a NonDifferentiableDecomposableFunctionType according to the static_assert error message.
In addition to the discrepancy between documentation and code, the static_assert error message speaks of decomposable function, but the documentation use the term separable function only. These terms are synonyms, but this may not be obvious to the user. If you ctrl+F in the documentation for decomposable, there is no result, which is unfortunate because it make fixing error in one's code more difficult. I suggest either saying the function need to be separable in the static_assert message, or mentioning the term decomposable function in the documentation.
http://ensmallen.org/docs.html
doc/functions_types.md
doc/optimizers.md
Mismatch on return types in PSO
../inst/include/ensmallen_bits/pso/pso.hpp: In member function ‘size_t ens::PSOType<VelocityUpdatePolicy, InitPolicy>::LowerBound() const’:
../inst/include/ensmallen_bits/pso/pso.hpp:162:38: error: cannot convert ‘const vec’ {aka ‘const arma::Col<double>’} to ‘size_t’ {aka ‘long unsigned int’} in return
162 | size_t LowerBound() const { return lowerBound; }
| ^~~~~~~~~~
Defined as arma::vec
:
ensmallen/include/ensmallen_bits/pso/pso.hpp
Lines 207 to 210 in b46986d
But returning size_t
:
ensmallen/include/ensmallen_bits/pso/pso.hpp
Lines 161 to 171 in b46986d
This is a port of mlpack/mlpack#404. There are some more details there. Here is the original description from @stephentu:
we need some strategy for dealing w/ a linearly dependent set of constraints. for LR-SDP, this isn't so big of a deal. but, this is especially key for the primal dual methods, which require this assumption so that various linear systems have unique solutions. we could either introduce a method which removes redundant constraints, or adapt the primal-dual method to handle degenerate linear systems.
note that, if we want the MVU SDP to work, then this will be necessary
To be coherent with mlpack vcpkg installation, I have made a PR to vcpkg project. I know ensmallen is a header-only library, but this way you install the header files in the folder Visual Studio looks for libraries automatically. To install it, if you have vcpkg installed, just type:
.\vcpkg install ensmallen:x64-windows
This will download, build and install the library with one command.
With ensmallen 1.13.0 adding SPSA
, there was a regression introduced. In particular, the SPSA
initialization of parameters is set to some unknown value. This was picked up under the -Wreorder
flag.
Details: https://stackoverflow.com/questions/1828037/whats-the-point-of-g-wreorder
Error Information:
* checking whether package 'RcppEnsmallen' can be installed ... WARNING
Found the following significant warnings:
../inst/include/ensmallen_bits/spsa/spsa.hpp:133:10: warning: 'ens::SPSA::Ak' will be initialized after [-Wreorder]
../inst/include/ensmallen_bits/spsa/spsa.hpp:123:10: warning: 'size_t ens::SPSA::maxIterations' [-Wreorder]
../inst/include/ensmallen_bits/spsa/spsa_impl.hpp:24:8: warning: when initialized here [-Wreorder]
See 'd:/RCompile/CRANguest/R-release/RcppEnsmallen.Rcheck/00install.out' for details.
Right now we have .cpp files in ensmallen_bits/problems/
; we could inline the methods and make them .hpp
files, which would let us keep the library as header-only. Any thoughts? If everyone agrees I can go ahead and do this.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.