What makes a coding problem hard for a model?
Which problem structures fool a model into producing a plausible but wrong solution?
I design adversarial coding tasks at Handshake AI to expose model failure modes. The goal is to find problems that are difficult for the right reason: a solver must reconcile evidence and apply the stated rules, rather than succeed by reproducing a familiar pattern. Comparing failed and successful difficulty methods is the useful result.
FindingFinite rule and convention tasks stayed at 2/2 solved in the selected local probes. Scope-conflict and archive-inference revisions reduced observed solves to 0/5 and 3/10.
Difficulty methods: what changed?
| Method / domain | Earlier → revised | Observed lesson |
|---|---|---|
| Finite rule enumeration · Formal verification | 2/2 → 2/2 | Still easy: a fully specified bounded search remained directly implementable. |
| Named convention profiles · CAD / fabrication | 2/2 → 2/2 | Still easy: fresh agents implemented the finite profile space even when old solutions failed. |
| Competing evidence scopes · Version control | 2/2 → 0/5 | Harder locally: plausible global processing violated an explicit scope boundary. |
| Archive-specific inference · ML / replay auditing | 2/2 → 3/10 | Harder locally after removing an undisclosed tie-break. The fair revision is reported, not the earlier 0/10. |
- Task & environment
- Reference / no-op checks
- Verifier review
- Agent trials
- Failure analysis
Easy to describe does not mean hard to solve
A dense specification can look difficult to a person while reducing to a small search or interpreter that an agent implements directly. In the selected formal-verification and CAD probes, adding role distinctions or convention rules still produced 2/2 fresh solves. Making an old candidate fail was not sufficient: fresh agents adapted.
Where the harder problems came from
The more promising methods made the agent determine which evidence applied, preserve scope boundaries, or infer behavior from the current archive instead of copying the visible example. A version-control redesign moved from 2/2 to 0/5 local solves. An ML replay-auditing task moved from 2/2 to 3/10 after a fairness correction. Several requirements changed between versions, so these are development observations rather than causal effect estimates.
Compare local probes with official evaluations
Local and official agents sometimes behaved differently. One analytics task had 0/5 local solves but 2/2 official solves on the recorded ledger architecture. That disagreement changed the development process: local runs diagnose failure mechanisms, while official evaluations measure the submitted task under that pipeline. The category chart keeps those official results separate from local redesign probes.
A failed model is only useful evidence if the task is fair
Some apparent difficulty came from an unstated tie-break or an ambiguous test. Those defects need repair, not credit as successful model failures. I use reference/no-op checks, verifier review and candidate regrading to distinguish a semantic miss from an evaluation defect. Timeouts remain unsuccessful attempts in the category chart, but are not automatically called reasoning failures.
What this research is building toward
The aim is a reusable taxonomy of failure-inducing problem structures, backed by examples of methods that worked and methods that did not. The current artifacts compare four selected local redesigns and every completed five-trial record in the inspected archive, spanning 11 official categories. A stronger comparison would hold model, budget and task variants fixed while varying the method, then repeat on new tasks. Private prompts, hidden tests and solutions remain outside the public data.