# Learning To Navigate The Synthetically Accessible Chemical Space Using Reinforcement Learning

**Authors:** Sai Krishna Gottipati, Boris Sattarov, Sufeng Niu, Yashaswi Pathak, Haoran Wei, Shengchao Liu, Karam M. J. Thomas, Simon Blackburn, Connor W. Coley, Jian Tang, Sarath Chandar, Yoshua Bengio  
**arXiv:** [2004.12485](https://arxiv.org/abs/2004.12485)  
**Date:** 2020-04-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2004.12485v2/

## Key points

- Machine learning for de novo drug design has made significant progress in the last decade, particularly in deep generative models.
- Current generative approaches face a major challenge as they do not ensure that the proposed molecular structures can be feasibly synthesized nor do they provide the synthesis routes of the proposed small molecules, which limits their practical applicability.
- A team of researchers has proposed a novel forward synthesis framework powered by reinforcement learning (RL) for de novo drug design called Policy Gradient for Forward Synthesis (PGFS).
- PGFS embeds the concept of synthetic accessibility directly into the de novo drug design system and enables an agent to navigate through synthetically accessible chemical space by subjecting commercially available small molecule building blocks to valid chemical reactions at every time step of the iterative virtual multi-step synthesis process.
- PGFS achieves state-of-the-art performance in generating structures with high QED and penalized clogP and was validated in an in-silico proof-of-concept associated with three HIV targets.
- The end-to-end training conceptualized in this study represents an important paradigm shift in radically expanding the synthesizable chemical space and automating the drug discovery process.
- PGFS offers a promising solution to one of the major challenges facing current generative approaches in de novo drug design. It opens up new possibilities for automating and accelerating drug discovery processes while significantly reducing costs.

## Summary

The field of machine learning for de novo drug design has made significant progress over the last decade, particularly in deep generative models. However, current generative approaches face a major challenge as they do not ensure that the proposed molecular structures can be feasibly synthesized nor do they provide the synthesis routes of the proposed small molecules, which limits their practical applicability. To address this challenge, a team of researchers has proposed a novel forward synthesis framework powered by reinforcement learning (RL) for de novo drug design called Policy Gradient for Forward Synthesis (PGFS). This framework embeds the concept of synthetic accessibility directly into the de novo drug design system. In this setup, an agent learns to navigate through the immense synthetically accessible chemical space by subjecting commercially available small molecule building blocks to valid chemical reactions at every time step of the iterative virtual multi-step synthesis process. The proposed environment for drug discovery provides a highly challenging test-bed for RL algorithms due to its large state space and high-dimensional continuous action space with hierarchical actions. PGFS achieves state-of-the-art performance in generating structures with high QED and penalized clogP. The team validated PGFS in an in-silico proof-of-concept associated with three HIV targets. Furthermore, the end-to-end training conceptualized in this study represents an important paradigm shift in radically expanding the synthesizable chemical space and automating the drug discovery process. The team describes how their approach could significantly reduce costs and accelerate drug development timelines by enabling more efficient exploration of chemical space. Overall, PGFS offers a promising solution to one of the major challenges facing current generative approaches in de novo drug design. By embedding synthetic accessibility into its framework and leveraging reinforcement learning techniques, it opens up new possibilities for automating and accelerating drug discovery processes.