laboroai / border Goto Github PK

View Code? Open in Web Editor NEW

33.0 33.0 6.0 1.89 MB

A reinforcement learning library in Rust

License: Apache License 2.0

Rust 90.00% Python 2.42% Jupyter Notebook 7.57%

reinforcement-learning rust

border's People

Contributors

Stargazers

Watchers

Forkers

lostmsu blofroth metavai sunnyboy00 taku-y

border's Issues

Add tests in examples

Add an associated type of configuration to Policy/Agent

Add a macro to implement Shape

Tweaks on example of vectorized environment for recent API changes

[v0.0.6] Fix bug in SimpleReplayBuffer::len()

Implements auto tuning of SAC alpha

[v0.0.6] Bump the version of tch-rs

Change ownership of this repository

The ownership of this repository will be changed to Laboro.AI, I'm working for. There will be no change on licenses of the crates in this repository.

Merge current changes on main branch, although not tied with the latest version of the crates
Transfer the ownership
Add a note about this change
Folk the repository after the transfer, because the crates before the version 0.0.5 are pointing to this repository in crates.io

Saving/loading configurations in files

Prioritized experience replay, reward clipping, double DQN

Train for Atari games other than Pong

Record environment's state in trainer

Save the model with the maximum evaluation reward during training

Rust native atari environments

Implements an environment that wraps atari-env. It does not require python interpreter to run.

Add an example of running a random policy on the environment.
Add the path to the ROM directory in the configuration of the environment
Add mapping from the name of each game and ROM file names
Test training on Pong with DQN
Add the random no-op steps when resetting
Add explanation for building atari-sys on MacOS

Randomness for policy in evaluation

Following the paper of DQN Atari, epsilon greedy policy with e=0.05 should be used in evaluation (Evaluation procedure, METHODS).

Modify the interface of SAC, making it similar with DQN and IQN

Correctly render the ground in PyBulletGym env

Add test for vectorized environment

[v0.0.6] Split functionality of ReplayBufferBase into two traits

This issue is created to split functionality of a replay buffer into two traits, one for pushing samples from environments, the other for making batches to train agents.

In addition, this issue removes the trait bound of Batch: BatchBase in ReplayBufferBase. This modification allows replay buffers implementing this trait to emit batches that does not have the form (o, a, o', r, is_done). Instead, agents should specify trait bounds for the type of samples taken from replay buffers.

Mitigating overestimation bias of critic in SAC

Set default log level to info in examples

Add String to RecordValue

Download a pertained model from google drive in some examples

Fix eval_with_recorder

The code below prevents to push the record at the final step. It should be simply removed.
https://github.com/taku-y/border/blob/7a28faa3a83538a0ce397f88c7624b344e60c58b/border-core/src/util.rs#L118-L120

[v0.0.6] Refactor dqn_cartpole example

The purpose of refactoring is to share code in example and test programs.

[v0.0.6] Hide public fields in `border_core::Trainer`

[v0.0.6] Fix SimpleReplayBuffer::len()

    fn len(&self) -> usize {
        if self.i < self.capacity {
            self.i
        } else {
            self.capacity
        }
    }

Replace the type of `is_done` from f32 to i8 in `Step` struct

Test SAC with GPU

Fails to build

Thanks for working on this crate, it seems to be the only deep RL crate available.
Unfortunately I'm getting errors when trying to build it:

D:\3rdparty\border> cargo run --example random_cartpole
   Compiling border v0.0.3 (D:\3rdparty\border)
error[E0308]: mismatched types
  --> src\env\py_gym_env\act_c.rs:65:65
   |
65 |                     let act = PyArrayDyn::<f32>::from_array(py, &act);
   |                                                                 ^^^^ expected struct `ndarray::ArrayBase`, found struct `ArrayBase`
   |
   = note: expected reference `&ndarray::ArrayBase<_, ndarray::dimension::dim::Dim<ndarray::dimension::dynindeximpl::IxDynImpl>>`
              found reference `&ArrayBase<OwnedRepr<f32>, Dim<IxDynImpl>>`
   = note: perhaps two different versions of crate `ndarray` are being used?

error[E0308]: mismatched types
  --> src\env\py_gym_env\act_c.rs:72:70
   |
72 |                         .map(|act| PyArrayDyn::<f32>::from_array(py, &act))
   |                                                                      ^^^^ expected struct `ndarray::ArrayBase`, found struct `ArrayBase`
   |
   = note: expected reference `&ndarray::ArrayBase<_, ndarray::dimension::dim::Dim<ndarray::dimension::dynindeximpl::IxDynImpl>>`
              found reference `&ArrayBase<ViewRepr<&f32>, Dim<IxDynImpl>>`
   = note: perhaps two different versions of crate `ndarray` are being used?

error[E0308]: mismatched types
  --> src\env\py_gym_env\act_c.rs:91:57
   |
91 |             let act = PyArrayDyn::<f32>::from_array(py, &act);
   |                                                         ^^^^ expected struct `ndarray::ArrayBase`, found struct `ArrayBase`
   |
   = note: expected reference `&ndarray::ArrayBase<_, ndarray::dimension::dim::Dim<ndarray::dimension::dynindeximpl::IxDynImpl>>`
              found reference `&ArrayBase<OwnedRepr<f32>, Dim<IxDynImpl>>`
   = note: perhaps two different versions of crate `ndarray` are being used?

error[E0308]: mismatched types
  --> src\env\py_gym_env\act_c.rs:98:62
   |
98 |                 .map(|act| PyArrayDyn::<f32>::from_array(py, &act))
   |                                                              ^^^^ expected struct `ndarray::ArrayBase`, found struct `ArrayBase`
   |
   = note: expected reference `&ndarray::ArrayBase<_, ndarray::dimension::dim::Dim<ndarray::dimension::dynindeximpl::IxDynImpl>>`
              found reference `&ArrayBase<ViewRepr<&f32>, Dim<IxDynImpl>>`
   = note: perhaps two different versions of crate `ndarray` are being used?

error[E0308]: mismatched types
   --> src\env\py_gym_env\framestack.rs:110:9
    |
104 |     fn get_ndarray(o: &PyAny) -> ArrayD<T2> {
    |                                  ---------- expected `ArrayBase<OwnedRepr<T2>, Dim<IxDynImpl>>` because of return type
...
110 |         o
    |         ^ expected struct `ArrayBase`, found struct `ndarray::ArrayBase`
    |
    = note: expected struct `ArrayBase<OwnedRepr<T2>, Dim<IxDynImpl>>`
               found struct `ndarray::ArrayBase<ndarray::data_repr::OwnedRepr<T2>, ndarray::dimension::dim::Dim<ndarray::dimension::dynindeximpl::IxDynImpl>>`
    = note: perhaps two different versions of crate `ndarray` are being used?

error[E0308]: mismatched types
  --> src\env\py_gym_env\obs.rs:43:33
   |
43 |                 obs.insert_axis(Axis(0))
   |                                 ^^^^^^^ expected struct `ndarray::dimension::axis::Axis`, found struct `Axis`
   |
   = note: perhaps two different versions of crate `ndarray` are being used?

error[E0308]: mismatched types
  --> src\env\py_gym_env\obs.rs:32:5
   |
26 |   pub fn pyobj_to_arrayd<S, T1, T2>(obs: PyObject) -> ArrayD<T2>
   |                                                       ---------- expected `ArrayBase<OwnedRepr<T2>, Dim<IxDynImpl>>` because of return type
...
32 | /     pyo3::Python::with_gil(|py| {
33 | |         let obs: &PyArrayDyn<T1> = obs.extract(py).unwrap();
34 | |         let obs = obs.to_owned_array();
35 | |         // let obs = obs.mapv(|elem| elem as f32);
...  |
48 | |         obs
49 | |     })
   | |______^ expected struct `ArrayBase`, found struct `ndarray::ArrayBase`
   |
   = note: expected struct `ArrayBase<OwnedRepr<T2>, Dim<IxDynImpl>>`
              found struct `ndarray::ArrayBase<ndarray::data_repr::OwnedRepr<T2>, ndarray::dimension::dim::Dim<ndarray::dimension::dynindeximpl::IxDynImpl>>`
   = note: perhaps two different versions of crate `ndarray` are being used?

error[E0308]: `if` and `else` have incompatible types
   --> src\env\py_gym_env\obs.rs:200:29
    |
190 | /                         if o.get_type().name().unwrap() == "NoneType" {
191 | |                             ArrayD::zeros(IxDyn(S::shape()))
    | |                             -------------------------------- expected because of this
192 | |                         }
193 | |                         // Processes the partial observation in the vectorized environment.
...   |
200 | |                             obs
    | |                             ^^^ expected struct `ArrayBase`, found struct `ndarray::ArrayBase`
201 | |                         }
    | |_________________________- `if` and `else` have incompatible types
    |
    = note: expected type `ArrayBase<OwnedRepr<_>, Dim<IxDynImpl>>`
             found struct `ndarray::ArrayBase<ndarray::data_repr::OwnedRepr<T2>, ndarray::dimension::dim::Dim<ndarray::dimension::dynindeximpl::IxDynImpl>>`
    = note: perhaps two different versions of crate `ndarray` are being used?

error: aborting due to 8 previous errors

For more information about this error, try `rustc --explain E0308`.
error: could not compile `border`

To learn more, run the command again with --verbose.

D:\3rdparty\border> rustc -V
rustc 1.55.0-nightly (240ff4c4a 2021-07-09)

Ant-v0 in pybullet-env

Fix building doc in docs.rs

Got the following error in docs.rs:

[INFO] [stderr]    Compiling torch-sys v0.3.1
[INFO] [stderr]     Checking image v0.23.14
[INFO] [stderr] error: failed to run custom build command for `torch-sys v0.3.1`
[INFO] [stderr] 
[INFO] [stderr] Caused by:
[INFO] [stderr]   process didn't exit successfully: `/opt/rustwide/target/debug/build/torch-sys-a53252c5a1353dd8/build-script-build` (exit code: 101)

Sparate py_gym_env and tch_agents into other crates

Make the interface of the model of DQN agent similar to that of IQN

Fails to create doc of border-py-eym-env 0.0.5

# build log
[INFO] running `Command { std: "docker" "create" "-v" "/home/cratesfyi/workspace/builds/border-py-gym-env-0.0.5/target:/opt/rustwide/target:rw,Z" "-v" "/home/cratesfyi/workspace/builds/border-py-gym-env-0.0.5/source:/opt/rustwide/workdir:ro,Z" "-v" "/home/cratesfyi/workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/home/cratesfyi/workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "DOCS_RS=1" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "3221225472" "--cpus" "3" "--user" "1001:1001" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:0cd99ca24d8e8c98e67c542213511d985b8778b5bdcbb160e038429496686047" "/opt/rustwide/cargo-home/bin/cargo" "+nightly" "rustdoc" "--lib" "-Zrustdoc-map" "-Z" "unstable-options" "--config" "build.rustdocflags=[\"-Z\", \"unstable-options\", \"--emit=invocation-specific\", \"--resource-suffix\", \"-20220124-1.60.0-nightly-51126be1b\", \"--static-root-path\", \"/\", \"--cap-lints\", \"warn\", \"--disable-per-crate-search\"]" "-Zunstable-options" "--config=doc.extern-map.registries.crates-io=\"https://docs.rs/{pkg_name}/{version}/x86_64-unknown-linux-gnu\"" "-j3" "--target" "x86_64-unknown-linux-gnu", kill_on_drop: false }`
[INFO] [stdout] 5f5e96f54646355243d606ed44d32a247a3e39db3428041c1144a6e489145a43
[INFO] [stderr] WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
[INFO] running `Command { std: "docker" "start" "-a" "5f5e96f54646355243d606ed44d32a247a3e39db3428041c1144a6e489145a43", kill_on_drop: false }`
[INFO] [stderr]     Checking numpy v0.14.1
[INFO] [stderr]    Compiling torch-sys v0.5.0
[INFO] [stderr]     Checking border-core v0.0.5
[INFO] [stderr] The following warnings were emitted during compilation:
[INFO] [stderr] 
[INFO] [stderr] warning: libtch/torch_api.cpp:1:9: fatal error: torch/csrc/autograd/engine.h: No such file or directory
[INFO] [stderr] warning:     1 | #include<torch/csrc/autograd/engine.h>
[INFO] [stderr] warning:       |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO] [stderr] warning: compilation terminated.
[INFO] [stderr] 
[INFO] [stderr] error: failed to run custom build command for `torch-sys v0.5.0`
[INFO] [stderr] 
[INFO] [stderr] Caused by:
[INFO] [stderr]   process didn't exit successfully: `/opt/rustwide/target/debug/build/torch-sys-e5ee8d1f43ffffa3/build-script-build` (exit status: 1)
[INFO] [stderr]   --- stdout
[INFO] [stderr]   cargo:rerun-if-env-changed=TORCH_CUDA_VERSION
[INFO] [stderr]   cargo:rerun-if-env-changed=LIBTORCH
[INFO] [stderr]   cargo:rustc-link-search=native=/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/torch-sys-0a078a45481dbdb4/out/libtorch/libtorch/lib
[INFO] [stderr]   cargo:rerun-if-changed=libtch/torch_api.cpp
[INFO] [stderr]   cargo:rerun-if-changed=libtch/torch_api.h
[INFO] [stderr]   cargo:rerun-if-changed=libtch/torch_api_generated.cpp.h
[INFO] [stderr]   cargo:rerun-if-changed=libtch/torch_api_generated.h
[INFO] [stderr]   cargo:rerun-if-changed=libtch/stb_image_write.h
[INFO] [stderr]   cargo:rerun-if-changed=libtch/stb_image_resize.h
[INFO] [stderr]   cargo:rerun-if-changed=libtch/stb_image.h
[INFO] [stderr]   cargo:rerun-if-env-changed=LIBTORCH_CXX11_ABI
[INFO] [stderr]   TARGET = Some("x86_64-unknown-linux-gnu")
[INFO] [stderr]   OPT_LEVEL = Some("0")
[INFO] [stderr]   HOST = Some("x86_64-unknown-linux-gnu")
[INFO] [stderr]   CXX_x86_64-unknown-linux-gnu = None
[INFO] [stderr]   CXX_x86_64_unknown_linux_gnu = None
[INFO] [stderr]   HOST_CXX = None
[INFO] [stderr]   CXX = None
[INFO] [stderr]   CXXFLAGS_x86_64-unknown-linux-gnu = None
[INFO] [stderr]   CXXFLAGS_x86_64_unknown_linux_gnu = None
[INFO] [stderr]   HOST_CXXFLAGS = None
[INFO] [stderr]   CXXFLAGS = None
[INFO] [stderr]   CRATE_CC_NO_DEFAULTS = None
[INFO] [stderr]   DEBUG = Some("true")
[INFO] [stderr]   CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
[INFO] [stderr]   running: "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/torch-sys-0a078a45481dbdb4/out/libtorch/libtorch/include" "-I" "/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/torch-sys-0a078a45481dbdb4/out/libtorch/libtorch/include/torch/csrc/api/include" "-Wl,-rpath=/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/torch-sys-0a078a45481dbdb4/out/libtorch/libtorch/lib" "-std=c++14" "-D_GLIBCXX_USE_CXX11_ABI=1" "-o" "/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/torch-sys-0a078a45481dbdb4/out/libtch/torch_api.o" "-c" "libtch/torch_api.cpp"
[INFO] [stderr]   cargo:warning=libtch/torch_api.cpp:1:9: fatal error: torch/csrc/autograd/engine.h: No such file or directory
[INFO] [stderr]   cargo:warning=    1 | #include<torch/csrc/autograd/engine.h>
[INFO] [stderr]   cargo:warning=      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO] [stderr]   cargo:warning=compilation terminated.
[INFO] [stderr]   exit status: 1
[INFO] [stderr] 
[INFO] [stderr]   --- stderr
[INFO] [stderr] 
[INFO] [stderr] 
[INFO] [stderr]   error occurred: Command "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/torch-sys-0a078a45481dbdb4/out/libtorch/libtorch/include" "-I" "/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/torch-sys-0a078a45481dbdb4/out/libtorch/libtorch/include/torch/csrc/api/include" "-Wl,-rpath=/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/torch-sys-0a078a45481dbdb4/out/libtorch/libtorch/lib" "-std=c++14" "-D_GLIBCXX_USE_CXX11_ABI=1" "-o" "/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/torch-sys-0a078a45481dbdb4/out/libtch/torch_api.o" "-c" "libtch/torch_api.cpp" with args "c++" did not execute successfully (status code exit status: 1).
[INFO] [stderr] 
[INFO] [stderr] 
[INFO] [stderr] warning: build failed, waiting for other jobs to finish...
[INFO] [stderr] error: build failed
[INFO] running `Command { std: "docker" "inspect" "5f5e96f54646355243d606ed44d32a247a3e39db3428041c1144a6e489145a43", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "5f5e96f54646355243d606ed44d32a247a3e39db3428041c1144a6e489145a43", kill_on_drop: false }`
[INFO] [stdout] 5f5e96f54646355243d606ed44d32a247a3e39db3428041c1144a6e489145a43

Support epsilon parameter of Adam

Support types other than f32 for ArrayD in PyGymEnvObs

[doc] Update document of border_core::eval

Test with vecenv fails on windows

Fix parameters of AtariWrapper when evaluating models

[v0.0.6] Add reset_with_index() in evaluation

If evaluates an agent several times, it would be nice to have methods to change configurations of the environment with an index, it may be a counter of evaluation loop during training.

[v0.0.6] Add method to create empty batch on trait Batch

Use Atari-ROM for testing in Github action

Got the error like this:

File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/atari_py/games.py", line 20, in get_game_path
509
    raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,))
510
Exception: ROM is missing for pong, see https://github.com/openai/atari-py#roms for instructions

Seems that manual install of the ROM is required in ci.yml.