Auto-recovery and failure modes

Overview

Once deliveries are flowing, WHAWIT does three things with the failures:

Groups them into recurring failure modes, ranked by the deliveries they cost — a prevention backlog.
Retries the ones a machine can fix, by calling an endpoint you register, under a policy you control.
Escalates everything else to a human, with the agent’s read on each one attached.

The order matters. Prevention shrinks the numerator, automation clears the mechanical remainder, and only what genuinely needs a decision reaches a person.

Failure modes

Every failed delivery carries an error signature — a normalized form of the message, with ids, timestamps, IP addresses and varying numbers collapsed. Failures sharing a signature are one failure mode. Unlike a log search, these persist and accumulate. A failure mode records how many times it has occurred, how many distinct deliveries it cost, which of your customers hit it, when it was first and last seen, and a daily trend.

Find them in the app under Reliability → Failure Modes, ranked by impact. The top row is the highest-value fix available.

Ranking

Impact weighs affected deliveries above raw occurrences — a customer-facing unit is worth more than a repeated log line — and scales the result by attribution and failure class. Your own platform faults outrank a customer’s misconfiguration, permanent failures outrank retryable ones, and a muted mode scores zero and leaves the backlog entirely. Each mode carries a status you move as you work it: open, accepted, fixed or muted, plus a link to the prevention ticket in your own tracker.

Classification

Each mode is classified on two axes:

transient | permanent | unknown

Would retrying help? A timeout or a 429 is transient. A validation error or a bad credential is permanent — retrying it only wastes the attempt.

platform | customer | upstream | unknown

Whose fault it was. Your platform, your customer’s configuration, or a third party you depend on.

Classification resolves strongest-first: a rule you wrote beats the model, and the model beats the built-in classifier. That order exists for a reason — “platform fault versus customer fault” is a business judgement, and no team should be asked to accept a verdict it cannot overrule.

API

Endpoint	Purpose
`GET /error-groups`	Ranked list. `rank` accepts `impact`, `recent` or `count`; filter by `status`, `failureClass`, `attribution`, `search`; paginated with `limit` and `page`.
`GET /error-groups/:urn`	One mode with its samples, trend, affected customers and recent deliveries.
`PATCH /error-groups/:urn`	Change status, override the classification, or link a prevention ticket.
`POST /error-groups/:urn/merge`	Two signatures, one bug.

Auto-recovery

WHAWIT does not reach into your systems. It calls an endpoint you register, and your endpoint decides how to re-drive the work.

How the loop closes

A delivery fails or misses its deadline

Both are the same decision. Silence past a deadline needs recovering just as much as a loud failure.

The policy is consulted

Does an enabled action match this failure? Is it under its attempt cap, its cooldown and its daily cap? Does it need a human to approve?

WHAWIT calls your endpoint

A signed request describing the delivery. A 2xx means the retry was accepted — nothing more.

Your retry emits normal outcome events

Whether the delivery actually recovered comes back the way everything else does. A later success flips it to auto-recovered, and it counts as hands-off in the KPI.

Or it escalates

Attempts exhausted, or the failure was never retryable — it goes to the on-call path.

A 2xx from your endpoint is not a recovered delivery. WHAWIT deliberately does not treat it as one: the only proof a delivery recovered is the delivery arriving.

What is never retried

Two guards, applied before the policy is even consulted:

Permanent failures. A retry cannot fix a malformed payload or a missing field.
Customer-attributed failures. A wrong credential in your customer’s configuration is not something another attempt resolves.

Retrying either burns the attempt cap and the cooldown, delaying the human who could actually fix it.

The policy

number

default:"3"

Hard ceiling on retries for one delivery.

number

default:"60"

Delay before the call goes out, giving a transient dip time to clear.

number

default:"120"

Minimum gap between two attempts on the same delivery.

number

default:"200"

Spend guard: attempts per action per UTC day.

boolean

default:"false"

Hold every trigger for a human decision.

low | medium | high | critical

Incidents strictly below this severity are auto-approved even when requiresApproval is set. Omit to require approval for everything.

number

default:"60"

How long a pending approval stays actionable.

Choosing what it answers

An action’s trigger narrows what it may act on: by failure class, by attribution, by workflow type, by specific failure modes, and by tenant kind. An empty list means “any”.

The request WHAWIT sends

Your endpoint receives a JSON body over HTTPS with two headers:

Header	Value
`x-whawit-signature`	`sha256=<hex>`
`x-whawit-timestamp`	Milliseconds since epoch

The MAC is computed over `${timestamp}.${rawBody}`, Stripe-style, so a captured request cannot be replayed later against a receiver that checks the age of the timestamp.

Verify the signature over the exact bytes you received, before parsing. Re-serialising the JSON changes key order and the check will fail.

import { createHmac, timingSafeEqual } from 'crypto';

function verify(rawBody: string, timestamp: string, presented: string, key: string): boolean {
  const mac = createHmac('sha256', key).update(`${timestamp}.${rawBody}`).digest('hex');
  const expected = Buffer.from(`sha256=${mac}`);
  const actual = Buffer.from(presented ?? '');
  if (expected.length !== actual.length) return false;
  return timingSafeEqual(expected, actual);
}

Reject requests whose timestamp is older than your tolerance — a minute or two is usually right.

Keys and endpoints

string

required

The id of a secret in GCP Secret Manager holding the signing key. The key itself never touches the database, source control or an environment file — you store an id, not a credential.

Endpoints must be HTTPS, and they are checked against an SSRF guard both when you save them and again at call time — loopback, private ranges, link-local and cloud metadata addresses are refused. Redirects are not followed: a 3xx from your endpoint is treated as a misconfiguration rather than chased, because following it would re-issue the signed payload at whatever address it named next.

API

Endpoint	Purpose
`GET /recovery-actions`	List actions for a project, with their stats.
`POST /recovery-actions`	Register one. Rejects an unsafe URL with a readable reason.
`GET`, `PATCH`, `DELETE /recovery-actions/:urn`	Read, update, remove.
`POST /recovery-actions/:urn/test`	Dry-run against your endpoint with a synthetic payload. Returns status, latency and a response snippet. Nothing is retried.
`GET /recovery-approvals?status=pending`	The approval inbox, including the exact payload that would be sent.
`POST /recovery-approvals/:urn/approve` \| `/reject`	Decide one.
`POST /deliveries/:urn/recover`	Trigger a retry by hand.
`POST /deliveries/:urn/escalate`	Hand a delivery to the on-call path.

A refusal from /recover is a 200 with status: "rejected" and a reason written for a person, not a 4xx. The reason is the point of the response; an error status would strip it out before anyone read it.

queued | pending_approval | rejected

queued is the only value that means a retry is actually going out.

string

Why the policy decided what it did.

Configuring it

Actions, their policy and the approval inbox live under Reliability → Recovery in the app. The stats on each action separate succeeded — your endpoint accepted the call — from recovered, the deliveries that actually came back green. The second is the number that matters.

The judgment queue

What survives prevention and automation is the residue: failures that need a decision rather than a mechanism. Reliability → Judgment Queue shows terminal deliveries that are still lost — automation either could not clear them or refused to try. Each one carries what failed and where, the classification, what recovery already attempted and why it stopped, and the AI agent’s read on the incident where one exists. Two actions resolve an item: retry it, or escalate it to a person. The agent works the same queue with the same two actions and, critically, under the same recovery policy — it cannot bypass an attempt cap or a cooldown, and it is told plainly when the policy refused rather than being allowed to believe a retry happened.

Reconciliation

A reliability number nobody can check is a number, not evidence. Reliability → Reconciliation imports your own independent record — a support export as CSV or JSON — and matches it against what WHAWIT recorded. Discrepancies are classified as missing in WHAWIT, a state mismatch, a timeliness mismatch, or missing in your source. Any one of them can become a tracked incident in a click. This is what makes the headline rate defensible: it survived a comparison with your own books.

Getting started

On-call

MCP Actions

Policies

Concepts

Delivery reliability

Developer Tools

Demos & resources

Company

Auto-recovery and failure modes

Overview

Failure modes

Ranking

Classification

API

Auto-recovery

How the loop closes

What is never retried

The policy

Choosing what it answers

The request WHAWIT sends

Keys and endpoints

API

Configuring it

The judgment queue

Reconciliation

​Overview

​Failure modes

​Ranking

​Classification

​API

​Auto-recovery

​How the loop closes

​What is never retried

​The policy

​Choosing what it answers

​The request WHAWIT sends

​Keys and endpoints

​API

​Configuring it

​The judgment queue

​Reconciliation

Overview

Failure modes

Ranking

Classification

API

Auto-recovery

How the loop closes

What is never retried

The policy

Choosing what it answers

The request WHAWIT sends

Keys and endpoints

API

Configuring it

The judgment queue

Reconciliation