Feature request: Elastic net about flux.jl

[ylxdzsw]

Since Lasso and Ridge models are linear models with L1 and L2 regularization, and ElasticNet is simply both, they can be implemented by a custom optimizer. To try this, first checkout this branch:

using Flux: Affine, train!, tobatch, Optimizer, WeightDecayConst, WeightDecayRatio, GradDecayFix, Accumulate

truth = Float32[0, 4, 2, 2, -3, 0, -1, 3, 2, -5]'

data = map(1:1024) do i
  x = rand(Float32, 10)
  x, truth * x + rand(Float32)
end

struct ElasticNet
  l1
  l2
end

Lasso(α) = ElasticNet(α, 0)
Ridge(α) = ElasticNet(0, α)

function fit!(m::ElasticNet, data)
  model = Affine(length(data[1][1]), 1)
  train!(model, data, epoch=50, opt=Optimizer([
    Accumulate(length(data) / 8),
    GradDecayFix(.002),
    WeightDecayRatio(m.l2),
    WeightDecayConst(m.l1)
  ]))
end

# train a lasso model
model = fit!(Lasso(5e-5), data)

# see the coefficients (for Lasso models, the 1st and 6th elements should be exactly 0)
@show model.W.x

# do prediction
@show model(tobatch(data[1][1]))[1], data[1][2]

SGD is not the best method to solve Lasso models, but nothing prevent it working. The optimizers currently not work with mxnet or tensorflow backend, but they will very soon.

[MikeInnes]

Closing this for now as I don't think there's an action item for Flux here; this should be pretty easy to build on top if needed (and of course I'd be happy to have it in the library if it's generally useful).