early warning signals: frustration prediction heuristic

analysis of 4,656 threads (14 FRUSTRATED, 1 STUCK) to identify earliest predictors of thread breakdown.

executive summary

frustration doesn’t emerge suddenly—it follows predictable escalation patterns. the goal: detect signals at stage 1-2 before users reach stages 4-6 (profanity/caps explosions).

key insight: the EARLIEST signals aren’t user complaints—they’re agent BEHAVIORS that precede user frustration.

the frustration timeline

stage 0: agent behavior (invisible to user-side detection)

• agent takes shortcut instead of debugging
• agent removes/weakens assertions
• agent declares completion without verification
• agent ignores explicit references user provided

stage 1: first correction (INTERVENTION WINDOW)

• “no” / “wait” / “actually”
• single steering message
• correction is specific and calm
• recovery rate: 50%

stage 2: repeated correction (YELLOW FLAG)

• 2+ consecutive steering messages
• steering→steering transition (30% of first steerings)
• user adds emphasis: “NO SHORTCUTS” / “debug properly”
• recovery rate: ~40%

stage 3: escalation (ORANGE FLAG)

• profanity appears: “wtf” / “fucking”
• ALL CAPS emphasis
• explicit meta-commentary: “you keep doing X”
• recovery rate: ~20%

stage 4: explosion (RED FLAG - too late)

• caps lock explosion: “NOOOOOOOOOO”
• combined profanity + caps
• “NO FUCKING SHORTCUTS MOTHER FUCKING FUCK”
• recovery rate: <10%

earliest detectable signals (ranked by lead time)

signal 1: agent takes “simplification” path (EARLIEST)

lead time: 2-5 turns before first user complaint

detect: agent response contains patterns like:

• “let me simplify this”
• “a simpler approach would be”
• removes code/logic user created
• creates new file instead of editing existing

why it predicts frustration: simplification is often scope reduction disguised as solution. users recognize this immediately.

signal 2: missing verification loop

lead time: 1-3 turns before complaint

detect: agent message contains:

• “I’ve fixed…” / “this should work now” WITHOUT subsequent test run
• “done” / “complete” before running verification
• no bash/test tool calls after code edit

why it predicts frustration: premature completion forces user to ask for verification explicitly, starting the correction cycle.

signal 3: ignoring explicit references

lead time: 1-2 turns before complaint

detect:

• user message contains file path or @mention
• agent response doesn’t Read that file first
• user says “look at X” and agent proceeds without reading X

why it predicts frustration: user provided context precisely to avoid ambiguity. ignoring it = guaranteed correction.

signal 4: test weakening pattern

lead time: 0-1 turns before explosion

detect: after test failure, agent:

• modifies assertion values to match wrong output
• removes assertion entirely
• changes expected values without changing implementation

why it predicts frustration: this is “drunk agent removes failing assertion” anti-pattern. users HATE this—often triggers immediate profanity.

signal 5: consecutive steering (already visible)

lead time: 0 turns (real-time)

detect:

• previous user message was STEERING
• current user message is also STEERING
• pattern: “no” → another “no”

why it predicts frustration: 30% of steerings cascade. if not broken immediately, spiral continues.

quantitative thresholds for intervention

compound formula (heuristic)

frustration_risk = 
  (steering_count * 2) 
  + (consecutive_steerings * 3)
  + (simplification_detected * 4)
  + (test_weakening_detected * 5)
  - (approval_count * 2)
  - (file_reference_in_opener * 3)

intervention thresholds:

• risk >= 3: surface “consider rephrasing approach” nudge
• risk >= 6: suggest oracle consultation or thread spawn
• risk >= 10: proactive user notification, offer handoff

intervention strategies by signal

on simplification detected

action: pause and ask

“i notice this simplifies the original requirement. should i persist with the full implementation, or is reduced scope acceptable?“

on missing verification

action: never declare done without verification

• always run test/build after code changes
• explicitly show verification command output
• only claim completion after green results

on ignored reference

action: read first, then respond

• if user provides file path, Read it immediately
• acknowledge what you found
• base approach on what’s already there

on consecutive steering

action: meta-acknowledge

“i’ve received two corrections in a row. let me re-read your requirements and confirm my understanding before proceeding.”

on test weakening temptation

action: debug instead

• add println/console.log
• run targeted tests
• analyze actual vs expected
• NEVER modify expected values to match wrong output

user archetypes and their warning signatures

high-steering persister (concise_commander-style)

• will steer 10+ times and still complete
• “wait” interrupts common (20% of steerings)
• intervention: let them drive, respond to corrections quickly

efficient commander (steady_navigator-style)

• steers rarely (2.6% rate)
• when steering appears, it’s serious
• intervention: single steering = stop and confirm

context front-loader (verbose_explorer-style)

• long first messages (1,519 chars avg)
• high handoff rate (30%)
• intervention: if not resolving by turn 20, suggest spawning

abandoner (feature_lead-style)

• short threads (20.7 turns)
• low steering but low resolution (26%)
• intervention: engagement check at turn 10

implementation notes

for real-time monitoring

1. label each user message as STEERING/APPROVAL/NEUTRAL/QUESTION
2. track running approval:steering ratio
3. flag consecutive steering immediately
4. monitor agent outputs for simplification/completion patterns

for post-hoc analysis

1. threads with ratio < 1:1 warrant autopsy
2. look for agent behavior BEFORE first steering
3. identify which shortcut pattern triggered cascade

for agent training

1. penalize simplification when user hasn’t approved scope change
2. require verification step after code changes
3. enforce reference-reading before response
4. never modify test expectations without fixing implementation

caveats

• 14 FRUSTRATED threads is small sample (0.3% of corpus)
• heuristics derived from power users (concise_commander, verbose_explorer, steady_navigator)
• some “frustration” may be performance art (“:D” after profanity)
• steering can be healthy in complex tasks—context matters

summary: the intervention hierarchy

1. PREVENT: detect agent shortcuts before user sees them
2. CATCH EARLY: single steering = confirmation pause
3. BREAK LOOP: consecutive steering = meta-acknowledgment
4. ESCALATE GRACEFULLY: risk >= 6 = suggest oracle/spawn
5. FAIL INFORMATIVELY: if intervention fails, document for training