andreacossu / continuallearning_recurrentnetworks

Evaluation of Continual Learning with Recurrent Neural Networks

Continual Learning with Recurrent Neural Networks

Evaluation of Continual Learning with Recurrent Neural Networks

To reproduce python launch_experiment_grid.py --config_file CONFIGS/path/to/config.yaml.

continuallearning_recurrentnetworks's Issues

Workers (tasks /actors) killed due to memory pressure (OOM)

Hello Andrea, I’m trying to work with this repository for a university project, I wanted to know if there were any requirements for PC components, infact running launch_exp_ssc_gem.py with speech_gem.yaml and launch_exp_grid.py with speech_replay.yaml I get the following error:

(raylet) [2023-12-31 11:58:01,756 E 1286] (raylet) node_manager.cc:3007: 1 Workers (tasks /actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at the node (ID: ..., IP: ...) in the last period of time. To see
more information about the workers killed on this node, use the logs ray raylet.out -ip ...
(raylet)
(raylet) Refer to the documentation on how to deal with the out-of-memory problem: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing the parallelism of tasks by requiring more CPU per task. To adjust the kill threshold, set the environment variable RAY_memory_usage_threshold when Ray starts. To disable worker killing, set the environment variable RAY_memory_monitor_refresh_ms to zero.

I also wanted to ask if the experiment type in the speech_gem.yaml of the lstm model is correct because it is not present in any of the possible configurations (attached image) of the run_exp() function in the launch_exp_ss_gem.py file, In addition, the ssc_esn function in line 47 has a declaration problem.

Waiting for your feedback thank you in advance.