03 / Operations tooling

Webhook Dashboard / Engineering experience

A webhook should leave more than a log line.

Follow signed synthetic events through verification, durable identity, alert reasoning, and operator investigation.

Enter the operations ledger

Synthetic signal / sanitized persisted

invoice.paid

14:32:08.214Z

Event ID: evt_demo_1042
Source: generic
Signature: valid
Alert state: fired · database only

Route

POST /api/webhooks/generic

01 / 07

Live event ledger

Selection commits the evidence under inspection.

These synthetic records model the repository’s persisted Event and AlertRun data. Invalid signatures do not appear here because the real route rejects them before persistence.

Persisted event ledger

Representative synthetic records · newest first

04 visible / static demo

02 / 07

Alert reasoning

An alert is a decision with evidence attached.

Inspect the rule condition, resulting operator action, and limitation. The third rule reflects tested optional Slack behavior rather than a claimed live integration.

Rule 01

Invoice paid events

genericdatabase only

Trigger condition: A generic event type begins with invoice.
Evidence used: source + event.type + lastFiredAt
Operator response: Inspect the event and its persisted AlertRun outcome.
Accepted limitation: Cooldown state is stored on one rule row; evaluation is synchronous.

Cooldown60 seconds

03 / 07

Operations topology

The ledger is the center of the diagnostic network.

Select a boundary to inspect what enters, what leaves, which signal it exposes, and where the present architecture stops.

Selected boundary

PostgreSQL ledger

Persists events, rules, and every alert evaluation for later inspection.

Data entering: Verified event + rule outcomes
Data leaving: durable operator evidence

Operational signal: Event and AlertRun records
Accepted tradeoff: Schema changes and migrations become operational work.

04 / 07

Failure investigation

Follow evidence until only one boundary remains suspect.

A guided Slack-delivery failure demonstrates how persisted context reduces debugging uncertainty without pretending automated remediation exists.

Representative failure scenario

This is a sanitized demonstration assembled from tested Slack-failure behavior—not a claimed production incident.

Investigation 01 / 05

Confirm trust

Observed signal

signatureValid: true

Evidence: The timestamped signature passed before parsing, so this is not an authentication rejection.
Working conclusion: The request crossed the public trust boundary successfully.

05 / 07

Operational decisions

Inspectable behavior starts with explicit contracts.

Each choice is tied to an implementation boundary, a nearby alternative, and a cost the system still carries.

Verify before parsing

JSON can be reformatted without changing its meaning, but signatures protect bytes.

Chosen approach: Read the raw request body and verify its source-specific HMAC before parsing or persistence.
Nearby alternative: Parse first and verify a reconstructed payload.
Accepted cost: The route must preserve exact raw-body semantics.

Give every accepted event one identity

Providers can deliver the same event more than once.

Chosen approach: Use a source-scoped external ID, or a deterministic payload hash when no ID exists, behind a unique database key.
Nearby alternative: Insert every delivery and reconcile later.
Accepted cost: Payload-hash identity depends on deterministic serialization when no provider ID exists.

Record every rule outcome

A missing notification does not reveal whether a rule failed to match, was disabled, or hit cooldown.

Chosen approach: Persist fired, skipped_cooldown, disabled, and no_match outcomes as AlertRun rows.
Nearby alternative: Store only alerts that fired.
Accepted cost: More rows are written, and retention policy is still future work.

Keep ingestion accepted when Slack fails

Turning an alert-channel outage into webhook retries can amplify duplicate traffic.

Chosen approach: Record slack_failed in the run note while preserving the accepted event response.
Nearby alternative: Return a provider-facing failure and request redelivery.
Accepted cost: There is no automatic delivery retry in the current implementation.

Reuse the evaluator for replay

A separate replay implementation can drift from ingestion behavior.

Chosen approach: Admin replay calls the same rule evaluator used by new events.
Nearby alternative: Build a replay-only execution path.
Accepted cost: Replay observes current rules and cooldown state, not a historical rules snapshot.

06 / 07

Technology evidence + limits

The implementation is useful because its boundaries are visible.

Technology is connected to concrete responsibility, then paired with what this repository does not claim.

Next.js 16

App Router pages, protected operations UI, webhook ingestion, health, and replay routes.

src/app + route handlers

Prisma + PostgreSQL

Unique event identity, JSON payload storage, rules, cooldown state, and AlertRun history.

prisma/schema.prisma

Clerk

Authenticated operator access and server-enforced admin authorization.

requireSessionUser / requireAdminUser

Zod

Server-side validation for alert-rule creation and updates.

alert action schemas

Vitest

Repeatable signature, ingestion, dedupe, replay, and Slack-delivery behavior.

tests/ + smoke script

Honest limitations

—The event stream shown here is clearly labeled representative synthetic data, not production traffic.
—Request limiting is held in process memory and is not shared across instances.
—Rule evaluation and optional Slack delivery run synchronously during ingestion.
—Only generic and stripe-like signature demonstrations are implemented.
—Slack is the only external alert adapter; failed delivery is recorded without an automatic retry queue.
—The repository makes no production-scale, uptime, traffic-volume, or guaranteed-delivery claim.

07 / 07

Reflection

Observability is part of the system, not a view added afterward.

The strongest lessons came from preserving context across trust, persistence, classification, and delivery boundaries.

Visibility / 01

Logs are not an operating surface.

A useful system joins the event, payload, identity, rule outcome, and delivery note where an operator can reason about them together.

Diagnosis / 02

Raw evidence narrows the fault boundary.

Signature state, deterministic identity, persistence, and alert history let an engineer rule out entire classes of failure before changing code.

Reliability / 03

A recorded failure is better than a hidden retry.

The current design keeps accepted ingestion stable when Slack fails, while being honest that durable retry is still a future boundary.

Evidence retained

The signal arrived. The reasoning stayed inspectable.

Experience hub Reopen the ledger