Why LLMs fail on prompts with a large number of requirements

The problem with complex prompts is not that a language model "fails to understand" individual instructions. It loses reliability when it has to satisfy many requirements at once: structure, format, safety constraints and details of content.

The study tested 15 models against 36 types of constraint — 369,753 checks in all. Procedurally generated tasks and deterministic checking code were used for evaluation, with no LLM judge involved.

The main result is unwelcome for the usual approach to prompts. As requirements are added, the quality of each individual constraint declines gradually. But the probability of satisfying all constraints at once collapses far faster: with eight requirements the model passed each of them in roughly 41% of cases, but all eight together in only 5.7%.

That is an important distinction. Average quality across individual items can look tolerable even though the finished answer almost always breaks at least one condition. The errors, moreover, are largely independent, so they accumulate multiplicatively. Sometimes the cause is simpler: one wrong property of the answer, an incorrect number of sentences say, immediately breaks several checks that rest on it.

Requirements are not equal either. Structural constraints — those tied to maintaining the shape of the answer and its sequence, for example — erode the model's original ability roughly twice as fast as lexical ones. A binary condition such as the presence of a specific word is easier to hold than a set of rules demanding constant tracking of structure and context.

Hence the practical conclusion: a long prompt with ten items is not one task but a bundle of independent risks. It is more reliable to split the work into several steps: first get the content, then transform it into the required structure, and afterwards run the result through separate checks. That does not make the model smarter, but it stops forcing it to hold every constraint in a single pass.

The result has limitations of its own. The test covers 15 models and from one to 12 simultaneous requirements, and all the checks are built procedurally. The study shows a general effect but does not tell you which particular model suits your prompt best. Besides, the threshold depends on the type of constraint: for the strongest model, success at the level of individual items drops below 50% already at seven requirements, and for 12 of the 15 models at three or fewer.

If your prompts regularly contain more than five or six mandatory conditions, do you already split them into stages or are you still counting on one perfect answer? Source: cs.AI updates on arXiv.org