RECAP: Regression evaluation for continual adaptation of prompts

Instruction-following benchmarks present the full constraint set at once, but deployed systems face evolving constraints that arrive incrementally and are rarely documented explicitly. Prompt-level adaptation methods exhibit forgetting and regression when constraints change over time. We introduce \methodname, a benchmark that measures continual-learning phenomena (forgetting, regression, and forward transfer) at the constraint level for prompt-level adaptation methods that keep model weights frozen. Built on the \recast dataset, \methodname applies typed add, edit, and delete operations under a proactive \emph{adapt-then-test} protocol: methods receive only the constraint specification and must integrate new requirements before seeing any test data, mirroring high-stakes deployment scenarios where systems must comply from the first interaction. We propose a decomposed evaluation framework with orthogonal metrics for regression (peak forgetting, collateral damage), edit uptake (four-case analysis, edit switch), and deletion (Unlearning Fidelity with three temporal variants). We evaluate six methods spanning no-adaptation baselines to evolutionary and Bayesian prompt optimizers across four LLMs (8B -- 120B parameters) and find that while such adaptation methods are helpful for offline settings or reactive online settings, they are not sufficient for the proactive, near real-time demands of production agentic systems where the first post-change interaction must already be compliant.