Adaptive instruction composition for automated LLM red teaming

Various approaches to LLM red teaming leverage an attacker LLM to discover jailbreaks against a target. Several of them task the attacker with formulating its own tactics to transform basic queries into jailbreaks through trial and error, resulting in a limited range of successful strategies. Another approach discovers diverse attacks by combining crowd-sourced queries and tactics within the attacker's instructions, but does so at random, limiting effectiveness. This article introduces a novel framework, Adaptive Instruction Composition, that represents instructions as a more generalized template and combines crowd-sourced texts according to an adaptive mechanism trained to jointly optimize attack effectiveness with diversity. We use reinforcement learning to balance exploration with exploitation in a combinatorial space of instructions to guide the attacker toward generations tailored to target vulnerabilities. We show that our strategy supplants a host of baselines, including random combinations, on the relevant metrics. To scale our framework to the massive space, we employ a neural contextual bandit that adapts to semantic features using contrastively pre-trained embeddings of individual texts. We anticipate that our framework can be used for a wide variety of generative applications.