Simplify Reinforcement Learning Agent Creation about mlpack

[tareknaser]

I have currently made these changes in my vscode, however I am not sure of the current syntax errors, since I dont know how to compile the entire mlpack library with my modifications.

I'd suggest you build mlpack main branch first. If you face any problems, feel free to open an issue. After you build mlpack, you should be able to follow RL tutorials
Then, add your modifications in a separate branch and build mlpack again to make sure the changes are working as expected.

is there a way I can give you the specific .hpp files where I have made the changes? or should I put a pull request so that my differences wrt the original code are visible?

You should open a PR.

[shapy051002]

(This is my first Open Source Attempt to help, I apologise if I have broken any protocol, and am open to suggestions to improve)

I have tried to implement the template as-

template <
typename EnvironmentType,
typename QNetworkType= decltype(std::declval().GetDefaultQNetwork(),
typename PolicyNetworkType= decltype(std::declval().GetDefaultPolicyNetwork()),
typename UpdaterType = AdamUpdate,
typename ReplayType = RandomReplay

and overloaded the TD3(ReplayType& replayMethod); constructor method in td3.hpp

in td3_impl.hpp, i have expanded upon the constructor
template <
typename EnvironmentType,
typename QNetworkType,
typename PolicyNetworkType,
typename UpdaterType,
typename ReplayType

TD3<
EnvironmentType,
QNetworkType,
PolicyNetworkType,
UpdaterType,
ReplayType

::TD3(ReplayType& replayMethod):
config(GetDefaultConfig()),
learningQ1Network(GetDefaultQNetwork()),
policyNetwork(GetDefaultPolicy()),
replayMethod(replayMethod),
//rest of the method remains the same

in training_config.hpp, i have implemented the GetDefaultConfig() method-
TrainingConfig GetDefaultConfig() const{
TrainingConfig config;
config.StepSize() = 0.01;
config.TargetNetworkSyncInterval() = 1;
config.UpdateInterval() = 3;
config.Rho() = 0.001;

}
and similarly for the other methods as well
Qlearning GetDefaultQNetwork() const{
FFN<EmptyLoss, GaussianInitialization>
qNetwork(EmptyLoss(), GaussianInitialization(0, 0.1));
QNetwork.Add(new Linear(128));
QNetwork.Add(new ReLU());
QNetwork.Add(new Linear(1));

}
GreedyPolicy GetDefaultPolicy() const{
FFN<EmptyLoss, GaussianInitialization>
policyNetwork(EmptyLoss(), GaussianInitialization(0, 0.1));
policyNetwork.Add(new Linear(128));
policyNetwork.Add(new ReLU());
policyNetwork.Add(new Linear(1));
policyNetwork.Add(new TanH());
}

I have currently made these changes in my vscode, however I am not sure of the current syntax errors, since I dont know how to compile the entire mlpack library with my modifications.

is there a way I can give you the specific .hpp files where I have made the changes? or should I put a pull request so that my differences wrt the original code are visible?

[5advaith]

This can also be done by using Wrapper Function or Class: A wrapper function or class that creates a TD3 agent with default parameters. Would be better right . It is safer than modifying the TD3 class itself. Will be more simple and elegant

[tareknaser]

Hey @5advaith
I am not familiar with this approach but could you open a PR where you implement it for just one reinforcement learning agent of your choice as a PoC?

cc: @zoq

[mlpack-bot]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions! 👍