Method __getstate__([]) does not exist & Training can't stop about rlgc

[qhuang-pnl]

@wuyuenhs The codes have been updated to make it compatible with the latest OpenAI Gym. Please try to install it in a new virtual environment (Don't use the yml file to create it as it is outdated). Please run the testPowerDynSimEnv_v3.py to make sure the environment is properly set up. Please report here if you have further issue.

[wuyuenhs]

Thanks for helping.
I follow the above to create a new env and install packages as needed. Then run test_PowerDynSimEnv_v3.py for testing. As shown below, it indicates that the environment is properly set up.

test completed
Connection with Ipss Server is closed

However, running trainIEEE39LoadSheddingAgent.py has the following error:

Traceback (most recent call last):
  File "trainIEEE39LoadSheddingAgent.py", line 144, in <module>
    train(ll, env, model_path)
  File "trainIEEE39LoadSheddingAgent.py", line 87, in train
    model = deepq.models.mlp([256,256])
AttributeError: module 'baselines.deepq.models' has no attribute 'mlp'

Baselines v0.1.6 removed def mlp. So I switched to baselines v0.1.4. Then error:

Traceback (most recent call last):
  File "trainIEEE39LoadSheddingAgent.py", line 144, in <module>
    train(ll, env, model_path)
  File "trainIEEE39LoadSheddingAgent.py", line 101, in train
    trained_model = model_path
TypeError: learn() got an unexpected keyword argument 'trained_model'

Changing the code of trainIEEE39LoadSheddingAgent.py can cause a series of uncontrollable errors. I don't know if these errors are caused by updates to the environment package(such as baselines). Of course, the latest version of OpenAI Gym is no problem.
If I return the other environment to the old version, there will be problems when I first asking questions in this issue.
So I don't know how to do it now.

[qhuang-pnl]

It turned out Baselines made some changes to the implementation. We fixed the issues with this update.

Our internal test environments are: OpenAI gym (0.15.3) and baselines (0.1.5)

Please pull the latest RLGC and try it again. Thanks again for your feedback, we really appreciate it.

[wuyuenhs]

Thanks to the update, the program is able to get the final Finished.
Unfortunately, it seems that there are some problems in the training process. In addition to FutureWarning, there are the following displays:

/root/anaconda3/envs/new/lib/python3.6/site-packages/numpy/core/fromnumeric.py:3257: RuntimeWarning: Mean of empty slice.
  out=out, **kwargs)
/root/anaconda3/envs/new/lib/python3.6/site-packages/numpy/core/_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
Saving final model to power_model_multistep_581_585_lr_5e-05.pkl
total running time is 5.142378330230713
Finished!!

I think the total running time is too short, I don't know if this is normal.
At the same time, the terminal that launch the Java server displays:

Observed states:
[voltageMag_Bus4, voltageMag_Bus7, voltageMag_Bus8, voltageMag_Bus18, voltageMag_Bus504, voltageMag_Bus507, voltageMag_Bus508, voltageMag_Bus518]
Initial values of the observed states:
[0.98712668745297, 0.9748013683313521, 0.9731456890757114, 1.0089329691108317, 0.9813836367392125, 0.9712347933931141, 0.9652967656365352, 1.005561904995773]
Bus, initLoad, shedLoadFraction = Bus504, 5.0, -0.2
Bus, initLoad, shedLoadFraction = Bus507, 2.338, -0.2
Bus, initLoad, shedLoadFraction = Bus518, 1.6320000000000001, -0.2
Bus, initLoad, shedLoadFraction = Bus507, 2.338, -0.2
Bus, initLoad, shedLoadFraction = Bus518, 1.6320000000000001, -0.2
Bus, initLoad, shedLoadFraction = Bus507, 2.338, -0.2
Bus, initLoad, shedLoadFraction = Bus518, 1.6320000000000001, -0.2
Bus, initLoad, shedLoadFraction = Bus507, 2.338, -0.2
Bus, initLoad, shedLoadFraction = Bus518, 1.6320000000000001, -0.2
Bus, initLoad, shedLoadFraction = Bus507, 2.338, -0.2
Bus, initLoad, shedLoadFraction = Bus518, 1.6320000000000001, -0.2
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : -5.551115123125783E-17
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : -5.551115123125783E-17
Bus, initLoad, shedLoadFraction = Bus507, 2.338, -5.551115123125783E-17
Bus, initLoad, shedLoadFraction = Bus518, 1.6320000000000001, -5.551115123125783E-17
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0

[qhuang-pnl]

Thanks for the confirmation. You can increase the total_timesteps in the deepq.learn() inside the train() function to increase the training, for example 500,000 or 1 Million steps.

The msgs in the java sever side are expected as the agent is mostly working under the exploration mode at the initial stage. The severe msgs mean the agent continues to try to shed the loads even there is no remaining load to shed.

[wuyuenhs]

Thank you for your tireless guidance. Now I can run the program successfully.
The RLGC-Project has taught me a lot. Thanks again！