Typical RL task tries to optimize the object function
$$U_T=U(R_1, R_2, ..., R_T)$$
where $R_i$ is the instaneous reward at time $t$. For general $Q$-learning or policy gradient algorithm, the object function is
$$U_T=\sum_{i=1}^{T}\gamma^{i-1}\cdot R_i$$
which is a discounted cunmulated sum of all one-step reward. However, such object function does not take market risk into consideration. The simplest benchmark to measure risk and return is the Sharpe ratio, i.e.
$$U_T=\frac{E(R)}{\text{std}(R)}$$
with $E(R)=(1/T)\sum_{i=1}^{T}R_i$ and std is the standard deviation. However, such object function is not additive, which prohibits us to use the old wisdom such as $Q$-learning and vanilla policy gradient methods.
J.Moody propose the idea of DSR, which can turn $U$ above into an additive sum of single step reward.
First, for given step n, the Sharpe ratio $U_n$ can be estimated by
Now if we expand the whole $U_T$ to first order of $\eta$, we have
$$U_T\simeq \eta \sum_{i=1}^T D_i + O(\eta^2)$$
Now the original object function of Sharpe ratio becomes totally additive to first order. One step reward is now replaced by $D_t$. It means that we can use $D_t$ as reward and leave all others of RL framework unchanged.