# Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning

**Authors:** Hyrum S. Anderson, Anant Kharkar, Bobby Filar, David Evans, Phil Roth  
**arXiv:** [1801.08917](https://arxiv.org/abs/1801.08917)  
**Date:** 2018-01-26  
**Summary page:** https://www.summarizepaper.com/en/paper/1801.08917v1/

## Key points

- Authors delve into signatureless malware detection using machine learning techniques
- Machine learning is effective in detecting unseen malware families and polymorphic strains
- Adversarial machine learning advancements highlight vulnerabilities of deep learning models and non-differentiable models
- Proposed framework based on reinforcement learning for attacking static PE anti-malware engines
- RL agent learns sequences of operations to evade detection for any given malware sample
- Experimental results demonstrate efficacy in attacking gradient-boosted machine learning models
- Attacks can potentially bypass components of publicly hosted antivirus engines
- Retraining models on evasive ransomware samples resulted in a 33% decrease in attack effectiveness
- Release of OpenAI gym by authors for researchers to analyze evasion rates against their own machine learning models, malware samples, and RL agents

## Summary

In their paper titled "Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning," authors Hyrum S. Anderson, Anant Kharkar, Bobby Filar, David Evans, and Phil Roth delve into the realm of signatureless malware detection using machine learning techniques. They highlight the effectiveness of machine learning in detecting previously unseen malware families and polymorphic strains, making it a valuable tool for primary detection engines and supplementary heuristic detection by anti-malware vendors. The authors discuss recent advancements in adversarial machine learning, pointing out that deep learning models are vulnerable to gradient-based attacks while non-differentiable models can be targeted by genetic algorithms aiming to reduce their scores systematically. To address these vulnerabilities, they propose a novel framework based on reinforcement learning (RL) for attacking static portable executable (PE) anti-malware engines. Unlike traditional approaches that rely on differentiable models or score-producing engines, the authors' framework equips an RL agent with functionality-preserving operations that can be applied to PE files. Through a series of strategic interactions with the anti-malware engine, the RL agent learns sequences of operations likely to evade detection for any given malware sample. This enables black-box attacks against static PE anti-malware systems and generates functional evasive malware samples as a direct outcome. Experimental results presented in the study demonstrate the efficacy of this method in attacking gradient-boosted machine learning models, showcasing substantial evasion rates that vary depending on the dataset used. The authors also show how attacks against these models can potentially bypass components of publicly hosted antivirus engines. Additionally, they explore adversarial training outcomes by retraining models on evasive ransomware samples resulting in a 33% decrease in attack effectiveness. Of significant note is the release of an OpenAI gym by the authors allowing researchers to analyze evasion rates against their own machine learning models, malware samples, and RL agents. This comprehensive framework opens up new avenues for studying and mitigating security threats posed by evolving malware variants through innovative reinforcement learning strategies.