# Reinforcement Learning for Supply Chain Attacks Against Frequency and Voltage Control

**Authors:** Amr S. Mohamed, Sumin Lee, Deepa Kundur  
**arXiv:** [2309.05814](https://arxiv.org/abs/2309.05814)  
**Date:** 2023-09-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.05814v1/

## Key points

- Modernization of power systems has led to new equipment installations and upgrades
- System vulnerability to malware through supply chain attacks
- Supply chain attacks allow cybercriminals to bypass network defenses and carry out deliberate attacks at the physical layer
- Cybercriminals are expected to leverage machine intelligence for sophisticated and adaptable attacks in supply chain attacks
- Reinforcement learning (RL) can be used to develop intelligent attacks in supply chain attacks against generation control devices
- RL-based malware can execute supply chain attacks and impact voltage and frequency regulation and stability
- Supply chain attacks on generation control devices like AVR, PSS, and governors discussed in Section II
- Formulation of supply chain attacks as an RL problem discussed in Section III
- Simulation of test cases on power system frequency and voltage stability presented in Section IV
- Findings highlight vulnerabilities in generation control devices and provide guidance for defending against supply chain attacks
- Reinforcement learning emphasizes the need for robust cybersecurity measures in power systems
- Understanding vulnerabilities and potential attack scenarios helps power system operators prepare against future threats leveraging machine intelligence.

## Summary

The modernization of power systems has brought about new equipment installations and upgrades. However, this has also made the system vulnerable to malware through supply chain attacks. These attacks pose a significant threat as they allow cybercriminals to bypass network defenses and carry out deliberate attacks at the physical layer. With the rapid advancements in machine intelligence, it is expected that cybercriminals will leverage this technology to create sophisticated and adaptable attacks that can be incorporated into supply chain attacks. In their study, the authors demonstrate how reinforcement learning (RL) can be used to develop intelligent attacks that can be incorporated into supply chain attacks against generation control devices. By simulating potential disturbances impacting frequency and voltage regulation, they showcase how RL-based malware can execute supply chain attacks and their potential impacts on voltage and frequency regulation and stability. The paper outlines the problem formulation in Section II where it discusses supply chain attacks in the context of compromising generation control devices such as Automatic Voltage Regulators (AVR), Power System Stabilizers (PSS), and governors. In Section III, the authors formulate supply chain attacks as an RL problem. They then proceed to simulate the impact of several test cases on power system frequency and voltage stability in Section IV. The findings from this study provide valuable guidance for defending against supply chain attacks by highlighting potential vulnerabilities in generation control devices. The demonstrated use of reinforcement learning for developing intelligent attacks emphasizes the need for robust cybersecurity measures in power systems. By understanding these vulnerabilities and potential attack scenarios, power system operators can better prepare themselves against future threats posed by cybercriminals leveraging machine intelligence. Overall, this research contributes to our understanding of how supply chain attacks can be executed using intelligent techniques like reinforcement learning. It underscores the importance of proactive cybersecurity measures in safeguarding critical infrastructure like power systems from evolving threats in an increasingly interconnected world.