At Wayfair I worked on a pipeline that built daily business reports from event streams. Kafka consumers picked up events, called other services to hydrate them, then wrote the result into CloudSQL Postgres on GCP.

Happy path was boring. Failures were not. Hydration APIs timed out. Consumers died mid-batch. Some payloads showed up missing fields we needed. Dropping those events was unacceptable for report accuracy. Blocking the whole consumer on one bad message was worse.

So we needed a Dead Letter Queue — a place to park failures, look at them, and replay later.

Why not a Kafka DLQ topic?

That was the default answer. It works if your only question is “did something fail?” It falls over when someone asks “what failed yesterday for event type X, and can I retry just those?” Kafka will move the bytes. Inspecting and filtering them means more consumers, more tooling, more waiting. We were already living in Postgres for the durable store. Putting failures in a table meant we could answer those questions with SQL at 2am.

So the DLQ became a Postgres table. Failed event → insert row with raw payload + why it failed. Status starts as PENDING. After a successful replay it becomes SUCCEEDED. Two states. That was enough.

Schema

CREATE TABLE dlq_events (
    id BIGSERIAL PRIMARY KEY,
    event_type VARCHAR(255) NOT NULL,
    payload JSONB NOT NULL,
    error_reason TEXT NOT NULL,
    error_stacktrace TEXT,
    status VARCHAR(20) NOT NULL, -- PENDING / SUCCEEDED
    retry_count INT NOT NULL DEFAULT 0,
    retry_after TIMESTAMP WITH TIME ZONE NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

• payload as JSONB — keep the original event, don’t invent a second schema
• retry_after — don’t hammer a dependency that is still down
• retry_count — hard stop without another datastore

CREATE INDEX idx_dlq_status
ON dlq_events (status);

CREATE INDEX idx_dlq_status_retry_after
ON dlq_events (status, retry_after);

CREATE INDEX idx_dlq_event_type
ON dlq_events (event_type);

CREATE INDEX idx_dlq_created_at
ON dlq_events (created_at);

The composite on (status, retry_after) is the one the scheduler actually cares about.

Retries without double-work

Visibility alone doesn’t replay anything. We ran a scheduled job with ShedLock so only one instance claimed the run across the fleet — no hand-rolled leader election.

dlq:
  retry:
    enabled: true
    max-retries: 240
    batch-size: 50
    fixed-rate: 21600000 # 6 hours

Every six hours, up to 50 eligible rows. Over max retries → skip. Success → SUCCEEDED. Still broken → stay PENDING for the next pass.

Claiming rows used FOR UPDATE SKIP LOCKED so two instances never processed the same failure:

@QueryHints(@QueryHint(name = "jakarta.persistence.lock.timeout", value = "-2"))
@Query(
    value = "SELECT * FROM dlq_table "
        + "WHERE messagetype = :messageType "
        + "AND retries < :maxRetries "
        + "AND (replay_status IS NULL OR replay_status NOT IN ('COMPLETED')) "
        + "ORDER BY created_at ASC "
        + "FOR UPDATE SKIP LOCKED",
    nativeQuery = true
)

Locked rows get skipped, not blocked. That is the whole trick.

Don’t fill the DLQ with flaky noise

Deadlocks, brief network blips, connection pool exhaustion — if every one of those went straight to the DLQ, the table would fill with messages that would have succeeded on a second try. That hides the real poison pills.

So the consumer retried in-process with exponential backoff before writing a DLQ row: start at 2s, double, cap at 30s, give up after three attempts.

ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(3);
backOff.setInitialInterval(2000L);
backOff.setMultiplier(2.0);
backOff.setMaxInterval(30000L);
backOff.setMaxAttempts(3);

Downstream can be dark for hours. Events pile up in the table. When the dependency comes back, the scheduler drains them. Bad events stay visible instead of vanishing into a topic nobody reads.

What I took away

Kafka stayed the ingestion pipe. Postgres became the failure workbook. We did not “replace” one with the other — we stopped asking Kafka to be a queryable ops UI.

Failure handling got dull. In production, dull is the goal.