Deconstructing instruction-following: A new benchmark for granular analysis of Large Language Model instruction compliance abilities
A modular framework that uses a dynamically generated dataset to evaluate the capability of Large Language Models.
Reliably ensuring Large Language Models (LLMs) follow complex instructions is a critical challenge, as existing benchmarks often fail to reflect real-world use or isolate compliance from task success. We introduce a modular framework that uses a dynamically generated dataset with up to 20 application-oriented generation constraints to enable a granular and independent analysis of this capability. Our evaluation of five LLMs from different families based on this new benchmark demonstrates that compliance is not a monolithic capability but varies significantly with constraint type, quantity, and position. The analysis reveals model-specific weaknesses, uncovers synergistic and conflicting interactions between instructions, and identifies distinct positional biases such as primacy and recency effects. These granular insights are critical for diagnosing model failures and developing more reliable LLMs for systems that demand strict adherence to complex instructions.
Latest publications
DF-RAG: Enhancing RAG for question answering by balancing relevance and diversity of retrieved chunks
A pipeline that dynamically adapts the level of diversity for each query at test time without requiring prior information.
EACLART: Adaptive Reasoning Trees for explainable claim verification
A hierarchical method for claim verification in Large Language Models.
EACLLessons from the field: An adaptable lifecycle approach to applied dialogue summarization
An industry case study on developing an agentic system to summarize multi-party interactions.
EACL