Comparing Stochastic Training Environments for Deep Reinforcement Learning with Deep Q-network: Learning Stability and Transfer to Real Data

Source abstract

The paper investigates how the choice of a generative stochastic model of the training environment affects the numerical stability of the learning and the transfer of the Double DQN policy to real time series. Four environments — GBM, GARCH(1,1)-t, Minimal Neural SDE, and Neural Jump-Diffusion — are compared, created on chronologically previous data of three financial time series: NVDA, SPY, and KO. For each combination of asset and environment, ten training runs are performed, a total of 120, with the architecture, state and action spaces, reward functions, and hyperparameters unchanged. The transfer was evaluated on a 250-day chronologically separated real window. Differences were found between the environments in all groups of indicators studied. Training in the GARCH environment was accompanied by the largest temporal-difference error and the largest gradient norms, but at the same time the smallest discrepancy of state space ranges and the lowest turnover on all three assets. In the Double DQN configuration used with Huber loss function and gradient clipping, no numerical divergence of estimates was observed in 120 runs; at the same time, the positive bias of Q-scores was preserved in all twelve asset–environment combinations. . In 14 of the 120 runs, the trained agent outperformed the corresponding fixed-allocation baseline selected on synthetic validation data. Agents generally held a larger share of capital in the asset when trained in environments with higher drift, although this pattern was not consistent across all assets. The obtained results characterize the engineering interaction between the generative environment and the DQN agent, but do not indicate the superiority of agents over passive deterministic control bases.