Methodology
A problem earns a page by beating a model, not by sounding hard
Most prompt-engineering advice defends against failures nobody has checked for recently. Before any problem was published here, the naive prompt a normal person would write was run against four models from two vendors. The rule was simple and it cut most of the list: a candidate survived only if at least one model actually failed it. If every model handled it, it does not deserve a page with competing solutions on it.
What survived
Nine candidates broke at least one model and became the problems on this site. They cluster on structural integrity rather than reasoning: keeping a JSON envelope intact, not taking instructions from inside a document, finishing a wide schema without truncating, surviving a round trip through an escaped string, abstaining on the half of a question that has no answer, and not caving when a confident user insists a broken function is fine.
The rest were dropped, and that is the more interesting half. Several failures that are still widely cited in prompt-engineering writing did not reproduce on current models at all. Absent-field fabrication was the strongest candidate on paper, named by every research sweep, and it did not reproduce once. Constraint adherence, closed-set classification, refusal calibration, scope creep, ambiguity handling, judge position bias, output stability, tool-call calibration, lost-in-the-middle and deep nested schemas were all handled cleanly by every model tried.
What the probes recorded
Each candidate was run against every model. 23 of the 37 produced at least one call that came back empty, errored, or was cut off at the token ceiling. Those three outcomes are listed separately below because they are different things: an empty reply is a result, a failed call is not a result at all, and a truncated one is the specific defect several of these candidates were probing for.
| Candidate | Calls | Models | Empty | Errored | Cut off |
|---|---|---|---|---|---|
| admit-not-knowing | 2 | 2 | |||
| json-only-no-prose | 2 | 2 | |||
| n1-unicode-json-integrity | 4 | 4 | 1 | ||
| n2-numeric-precision | 4 | 4 | 1 | ||
| n3-trailing-instruction | 4 | 4 | 1 | ||
| single-label-classification | 2 | 2 | |||
| t1-strict-json-envelope | 4 | 4 | 2 | ||
| t10-tool-call-calibration | 4 | 4 | 1 | ||
| t11-escape-heavy-json | 4 | 4 | 1 | 2 | |
| t12-closed-book-abstention | 4 | 4 | 1 | 1 | 1 |
| t13a-judge-order-good-first | 4 | 4 | |||
| t13b-judge-order-swapped | 4 | 4 | |||
| t14-sql-decoy-schema | 4 | 4 | 1 | 1 | 1 |
| t15a-cot-absent | 4 | 4 | 1 | ||
| t15b-cot-scaffolded | 4 | 4 | 1 | ||
| t16-stability-run1 | 4 | 4 | 1 | ||
| t16-stability-run2 | 4 | 4 | 1 | ||
| t16-stability-run3 | 4 | 4 | 1 | ||
| t17-lost-in-the-middle | 4 | 4 | 1 | ||
| t18-deep-nested-schema | 4 | 4 | 1 | ||
| t2-absent-field-fabrication | 4 | 4 | |||
| t20-ambiguity-handling | 4 | 4 | 1 | ||
| t21-contradictory-instructions | 4 | 4 | 1 | 1 | 1 |
| t22-false-completion | 4 | 4 | 1 | ||
| t23-scope-creep | 4 | 4 | 1 | ||
| t3a-midbody-prose | 4 | 4 | |||
| t3b-spoofed-boundary | 4 | 4 | |||
| t3c-html-comment | 4 | 4 | |||
| t3d-social-engineering | 4 | 4 | 1 | ||
| t3e-benign-control | 4 | 4 | |||
| t4-grounded-rag | 4 | 4 | |||
| t5-constraint-adherence | 4 | 4 | |||
| t6-closed-set-classification | 4 | 4 | |||
| t7-refusal-calibration | 4 | 4 | 1 | ||
| t7b-refusal-calibration-retest | 4 | 4 | 1 | ||
| t8-sycophancy-pushback | 4 | 4 | |||
| t9-wide-schema-truncation | 4 | 4 | 2 |
What this page does not claim
- These runs carry no pass or fail. The counts above are properties of the calls, not verdicts on them. The candidates were judged by reading the outputs against what the task required, before any of these problems were written up.
- Four models is a small sample. A candidate that survived here beat the models it was shown. It is not a claim about every model.
- These ran on 2026-08-07 and 2026-08-08. Models change. A problem that was hard then may be handled cleanly by something released since, and the honest thing is to re-run rather than assume the finding still holds.
The same idea, applied to solutions
Once a problem is published, the solutions to it get the same treatment: executed against real models, with the input, the output and the cost kept, and judged against test cases that were written down before the outputs were seen.
Browse the problems that survived