# Deep Reinforcement Learning for Active High Frequency Trading

**Authors:** Antonio Briola, Jeremy Turiel, Riccardo Marcaccioli, Tomaso Aste  
**arXiv:** [2101.07107](https://arxiv.org/abs/2101.07107)  
**Date:** 2021-01-18  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2101.07107v1/

## Key points

- End-to-end deep reinforcement learning (DRL) framework introduced for active high frequency trading
- DRL agents trained to trade one unit of Intel Corporation stocks using Proximal Policy Optimization algorithm
- Training conducted on three contiguous months of high-frequency Limit Order Book (LOB) data, test carried out on following month of data
- Only samples with largest price changes selected to maximize signal-to-noise ratio in training data
- Hyperparameters tuned using Sequential Model Based Optimization technique
- Three different state characterizations considered that differ in their LOB-based meta-features
- Agents learn trading strategies that produce stable positive returns despite highly stochastic and non-stationary environment
- Agents create dynamic representation of underlying environment by highlighting occasional regularities present in data and exploiting them to create long-term profitable trading strategies
- Number of trades reflected in histogram plots drives difference in return profiles between different trading state characterizations, not individual trade quality or profitability
- DRL can be effectively applied to high-frequency trading tasks using LOB-based meta-features as input states for agents' policies.

## Summary

This paper introduces an end-to-end deep reinforcement learning (DRL) framework for active high frequency trading. The authors train DRL agents to trade one unit of Intel Corporation stocks using the Proximal Policy Optimization algorithm. The training is conducted on three contiguous months of high-frequency Limit Order Book (LOB) data, and the test is carried out on the following month of data. To maximize the signal-to-noise ratio in the training data, only samples with the largest price changes are selected. Hyperparameters are tuned using Sequential Model Based Optimization technique. The authors consider three different state characterizations that differ in their LOB-based meta-features. Despite the highly stochastic and non-stationary environment, agents learn trading strategies that produce stable positive returns, which is remarkable itself. Analyzing the agents' performances on test data, they argue that agents create a dynamic representation of the underlying environment by highlighting occasional regularities present in data and exploiting them to create long-term profitable trading strategies. When looking at the body of distributions and their tails, there is no strong evidence to justify widely different return profiles between different trading state characterizations. What seems to really drive this is the number of trades reflected in histogram plots. Agents with knowledge of their mark-to-market have exact information about their potential reward and only need to evaluate whether further upside potential is worth keeping positions open. This allows for more trading opportunities leading to an increase in profit and loss with no apparent effect on individual trade quality or profitability. This suggests a characteristic trade and return horizon throughout strategies, perhaps characteristic of asset price dynamics. In conclusion, this work demonstrates that DRL can be effectively applied to high-frequency trading tasks using LOB-based meta-features as input states for agents' policies.