Question about the definition of reward() about mpopis

[dylan-asmar]

Sorry for the delay in response.

For the CarRacingEnv, the state space is the same as the observation space defined here. The reward function you linked (line 203) takes a single environment and returns a single reward. Since the velocity components are the 4th and 5th entries of the state vector, it uses env.state[4:5] for the velocity.

In the calculate_trajectory_costs function, since we haveK samples we create K different environments and simulate them across T time steps (here). The control cost is a scalar value in line 204 as it is from sample k and time step t.

After stepping through all the K samples for the T time steps, we have a trajectory cost for each sample which is of size K.

So the reward function returns the reward based on the environment at a given state (contained within the environment struct). The calculate_trajectory_costs function calculates the cost of each K samples across the time horizon by calling the reward function at each time step for each sample (and combining it with the control cost).

The function defined here

MPOPIS/src/mppi_mpopi_policies.jl

Line 186 in ceff32b

function calculate_trajectory_costs(pol::MPPI_Policy, env::AbstractEnv)

Is the main function for calculating the trajectory costs for each sample for most environments that are a subtype of AbstractEnv. The function defined here

MPOPIS/src/mppi_mpopi_policies.jl

Line 148 in ceff32b

function calculate_trajectory_costs(pol::MPPI_Policy, env::EnvpoolEnv)

is the same function, but for use with the EnvpoolEnv environment. This function is for MuJoCo environments and uses EnvPool to help with running numerous MuJoCo simulations at once. In this function, reward(env) returns a vector of size K which is the number of environments. So we only need to loop over the T time steps in this function.

I hope this helps clear up some of the confusion. Let me know if you have any more questions.

[dylan-asmar]

Hello!

The code was structured to allow for extensions of other environments based on CommonRLInterface. The reward for each environment are extensions of the reward function defined in CommonRLInterface.

Code location for the reward function for different environments:

• CartPole: I did not redefine the reward function and used the one implemented in ReinforcementLearning.jl here
• • MountainCar:

    MPOPIS/src/examples/mountaincar_example.jl

    Line 14 in ceff32b

    function RLBase.reward(env::MountainCarEnv{A,T}) where {A,T}

• CarRacing:

  MPOPIS/src/envs/car_racing.jl

  Line 203 in ceff32b

  function RLBase.reward(env::CarRacingEnv{T}) where {T}

• MultiCarRacing:

  MPOPIS/src/envs/multi-car_racing.jl

  Line 138 in ceff32b

  function RLBase.reward(env::MultiCarRacingEnv{T}) where {T}

• MuJoCo: It is defined using the default reward from the Python implementation. and is defined here

  MPOPIS/src/envs/envpool_env.jl

  Line 105 in ceff32b

  RLBase.reward(env::EnvpoolEnv) = env.rews

From Line 174 that you linked, that is how we are calculating Line 9 in the pseudocode. We are just incrementally adding the costs across all time steps. reward(env) is the state-dependent cost at each time step and returns the terminal cost at the final time step. So it represents φ(X) + c(X) from the pseudocode. Line 167 is where the control costs are being calculated which is the third term in Line 9 from the pseudocode in the paper.

I hope this helps clear up any confusion.

[dylan-asmar]

Let me know if you still have any questions about this.

[YihanLi126]

Hello!

The code was structured to allow for extensions of other environments based on CommonRLInterface. The reward for each environment are extensions of the reward function defined in CommonRLInterface.

Code location for the reward function for different environments:

• CartPole: I did not redefine the reward function and used the one implemented in ReinforcementLearning.jl here
• • MountainCar:

    MPOPIS/src/examples/mountaincar_example.jl

    Line 14 in ceff32b

    function RLBase.reward(env::MountainCarEnv{A,T}) where {A,T}

• CarRacing:

  MPOPIS/src/envs/car_racing.jl

  Line 203 in ceff32b

  function RLBase.reward(env::CarRacingEnv{T}) where {T}

• MultiCarRacing:

  MPOPIS/src/envs/multi-car_racing.jl

  Line 138 in ceff32b

  function RLBase.reward(env::MultiCarRacingEnv{T}) where {T}

• MuJoCo: It is defined using the default reward from the Python implementation. and is defined here

  MPOPIS/src/envs/envpool_env.jl

  Line 105 in ceff32b

  RLBase.reward(env::EnvpoolEnv) = env.rews

From Line 174 that you linked, that is how we are calculating Line 9 in the pseudocode. We are just incrementally adding the costs across all time steps. reward(env) is the state-dependent cost at each time step and returns the terminal cost at the final time step. So it represents φ(X) + c(X) from the pseudocode. Line 167 is where the control costs are being calculated which is the third term in Line 9 from the pseudocode in the paper.

I hope this helps clear up any confusion.

Yes, the explanation is clear for me, and thank you for your reply!
I'll let you know if I have any questions in my following work.

[YihanLi126]

Hello!
I have a small question about the dimension of the reward:

MPOPIS/src/envs/car_racing.jl

Line 203 in ceff32b

function RLBase.reward(env::CarRacingEnv{T}) where {T}

I'm a little bit confused about the I/O of the state of the environment. It seems that the function here uses env.state[4:5] for calculating the cost of the velocity. Is it just for a single state, or a series of states?(That is to say, what is the data structure of the state of the environment?) I have this question because both trajectory_cost and control_costs here are array with K elements:

MPOPIS/src/mppi_mpopi_policies.jl

Line 174 in ceff32b

trajectory_cost = trajectory_cost - reward(env) + control_costs

if the reward function calculates the cost of a series of state costs, where can I find function that apply the system dynamics to get the the state related to the input? Is it like the step function here:

MPOPIS/src/envs/car_racing.jl

Line 284 in ceff32b

function _step!(env::CarRacingEnv, a::Vector{Float64})

And what are the purposes of the two cost function here for two kinds of environments:

MPOPIS/src/mppi_mpopi_policies.jl

Line 148 in ceff32b

function calculate_trajectory_costs(pol::MPPI_Policy, env::EnvpoolEnv)

MPOPIS/src/mppi_mpopi_policies.jl

Line 186 in ceff32b

function calculate_trajectory_costs(pol::MPPI_Policy, env::AbstractEnv)

Can I get some information about differences between them which served for different purposes?

Thank you for your patience!