$\mathsf {QFlip}$: An Adaptive Reinforcement Learning Strategy for the $\mathsf {FlipIt}$ Security Game

Abstract

A rise in Advanced Persistent Threats (APTs) has introduced a need for robustness against long-running, stealthy attacks which circumvent existing cryptographic security guarantees. $\mathsf {FlipIt}$ is a security game that models attacker-defender interactions in advanced scenarios such as APTs. Previous work analyzed extensively non-adaptive strategies in $\mathsf {FlipIt}$, but adaptive strategies rise naturally in practical interactions as players receive feedback during the game. We model the $\mathsf {FlipIt}$ game as a Markov Decision Process and introduce $\mathsf {QFlip}$, an adaptive strategy for $\mathsf {FlipIt}$ based on temporal difference reinforcement learning. We prove theoretical results on the convergence of our new strategy against an opponent playing with a Periodic strategy. We confirm our analysis experimentally by extensive evaluation of $\mathsf {QFlip}$ against specific opponents. $\mathsf {QFlip}$ converges to the optimal adaptive strategy for Periodic and Exponential opponents using associated state spaces. Finally, we introduce a generalized $\mathsf {QFlip}$ strategy with composite state space that outperforms a Greedy strategy for several distributions including Periodic and Uniform, without prior knowledge of the opponent’s strategy. We also release an OpenAI Gym environment for $\mathsf {FlipIt}$ to facilitate future research.