tspooner / lfa

Native Rust implementations of linear function approximators.

LFA (api)

Overview

lfa provides a set of implementations for common linear function approximation techniques used in reinforcement learning.

Installation

[dependencies]
lfa = "0.15"

Note that lfa enables the blas feature of its ndarray dependency, so if you're building a binary, you additionally need to specify a BLAS backend compatible with ndarray. For example, you can add these dependencies:

blas-src = { version = "0.2.0", default-features = false, features = ["openblas"] }
openblas-src = { version = "0.6.0", default-features = false, features = ["cblas", "system"] }

See ndarray's README for more information.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate and adhere to the angularjs commit message conventions (see here).

License

MIT

lfa's Issues

Replace macro trait implementations with procedural macros (Rust 2018)

Add state preprocessing module

Handling infinite dimensional domains can be problematic. In many reinforcement learning experiments, for example, the engineer may want to perform extensive preprocessing to the environment state before passing it off to the function approximator.

The answer is to add functionality for handling this explicitly with, ideally, a futures-like pipeline structure; this could be a good opportunity to start incorporating futures in the framework. Of course, it should be opt-in, i.e. the code should work with or without a preprocessor.

Unified approximator construction

At the moment each Approximator type has it's own constructor methods. This should be unified such that there is a standard method for constructing approximators on input spaces: from_space. This will require variable arguments since each instance has it's own unique parameters that are required as input.

One possible approach might be something like the following:

pub trait FromSpace<S: Space> {
    fn from_space<A>(input_space: S, args: A) -> Self;
}

This may take a bit of work and should be thought through properly. Otherwise this problem will just be revisited over and over again.

Adaptive function approximation architectures

At the moment, linear function approximation in rsrl requires stationary feature vectors. However, a lot of research conducted in the area of feature-based representations suggest that learning dependencies online can produce superior results. For example:

1. Sparse Distributed Memories (SDM)
2. Adaptive Tile Coding (ATC)
3. Incremental Feature Dependency Discovery (iFDD)
4. Bellman-error Basis Functions (BEBF)
5. Orthogonal Matching Pursuit (OMP)

To do this we need either a new class, perhaps called AdaptiveLinear, which handles changes in the weight vector or just custom linear function approximators for each of the above.