Skip to main content

Sequencer Operations

The ingest pipeline

Every event passes through the same ordered stages. Knowing the order tells you where a rejection came from:
A rejection at dedupe is usually a client retry working correctly, not an error. A rejection at verify signature on previously-working traffic usually means a key rotation that didn’t complete.

Batch lifecycle

  1. Events are accepted and assigned sequence numbers within their stream.
  2. A batch closes and its Merkle tree is built over event leaves.
  3. A commitment is emitted — prev_state_root, new_state_root, events_root, sequence range.
  4. The commitment is anchored on Set L2.
  5. Receipts and inclusion proofs become available to clients.
Steps 3 and 4 are distinct, and the gap matters. After step 3 you have a sequenced event; only after step 4 is it verifiable by a third party. A receipt issued between the two proves sequencing, not settlement.

What to monitor

Proof-conflict counters are worth alerting on specifically — ves_compliance_proof_conflict, ves_compliance_proof_public_inputs_conflict, and ves_validity_proof_conflict. A conflict means two proofs disagree for the same event, which is a correctness signal rather than a capacity one.

Dead letters

Events the sequencer permanently rejected land in a dead-letter queue rather than vanishing. Treat non-zero depth as a real alert: something upstream is producing events the pipeline won’t accept, and every one of them is missing from the anchored history.

Anchor lag is the one that matters

Ingest and batching can be healthy while anchoring is stalled. When that happens, events keep sequencing and nothing is externally verifiable — which is invisible until someone tries to verify.
Alert on anchor lag independently of ingest health.

Health probes

Use /ready for load-balancer membership; /health alone will keep a sequencer in rotation while it can’t reach Postgres.

Admin surface