Webhook system with logging and retry functionality

Setting Up Webhook System with Logging and Retry

A webhook is an outgoing HTTP request that you can't fully control. The recipient may return 200 without processing data. It may fail 9 seconds after receiving. It may silently drop the event. Without detailed logging of all attempts and the ability to manually resend, debugging integration issues is nearly impossible.

What to Log Per Attempt

Minimum dataset per delivery attempt:

Attempts Table Schema

CREATE TABLE webhook_attempts (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  delivery_id UUID NOT NULL REFERENCES webhook_deliveries(id) ON DELETE CASCADE,
  attempt_number INTEGER NOT NULL,
  started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  duration_ms INTEGER,
  request_body JSONB,
  request_headers JSONB,
  response_code INTEGER,
  response_headers JSONB,
  response_body TEXT,           -- truncated to 2000 characters
  error_message TEXT,
  success BOOLEAN NOT NULL DEFAULT false
);

CREATE INDEX idx_attempts_delivery ON webhook_attempts(delivery_id);
CREATE INDEX idx_attempts_started ON webhook_attempts(started_at DESC);

Store attempts separately from deliveries—one delivery can have 8 attempts. This lets you see full history and understand where things went wrong.

Logging Implementation

class WebhookAttemptLogger
{
    public function log(
        WebhookDelivery $delivery,
        int $attempt,
        WebhookAttemptData $data
    ): WebhookAttempt {
        return WebhookAttempt::create([
            'delivery_id'      => $delivery->id,
            'attempt_number'   => $attempt,
            'started_at'       => $data->startedAt,
            'duration_ms'      => $data->durationMs,
            'request_body'     => $delivery->payload,
            'request_headers'  => $data->requestHeaders,
            'response_code'    => $data->responseCode,
            'response_headers' => $data->responseHeaders,
            'response_body'    => $data->responseBody
                ? mb_substr($data->responseBody, 0, 2000)
                : null,
            'error_message'    => $data->errorMessage,
            'success'          => $data->success,
        ]);
    }
}

class SendWebhookJob implements ShouldQueue
{
    public function handle(
        WebhookAttemptLogger $logger
    ): void {
        $startedAt = now();
        $requestHeaders = $this->buildHeaders();

        try {
            $response = Http::timeout(15)
                ->withHeaders($requestHeaders)
                ->post($this->delivery->subscription->endpoint_url, $this->delivery->payload);

            $durationMs = (int)(microtime(true) * 1000 - $startedAt->timestamp * 1000);

            $logger->log($this->delivery, $this->delivery->attempt_count, new WebhookAttemptData(
                startedAt: $startedAt,
                durationMs: $durationMs,
                requestHeaders: $requestHeaders,
                responseCode: $response->status(),
                responseHeaders: $response->headers(),
                responseBody: $response->body(),
                success: $response->successful(),
            ));

            if ($response->successful()) {
                $this->delivery->markDelivered();
            } else {
                $this->delivery->scheduleRetry();
            }

        } catch (\Throwable $e) {
            $durationMs = (int)(microtime(true) * 1000 - $startedAt->timestamp * 1000);

            $logger->log($this->delivery, $this->delivery->attempt_count, new WebhookAttemptData(
                startedAt: $startedAt,
                durationMs: $durationMs,
                requestHeaders: $requestHeaders,
                errorMessage: get_class($e) . ': ' . $e->getMessage(),
                success: false,
            ));

            $this->delivery->scheduleRetry();
        }
    }
}

Manual Resend

An admin or developer should be able to resend any event without code changes. This is critical for debugging integrations and recovery after failures.

class WebhookDeliveryController extends Controller
{
    // Resend specific delivery
    public function resend(WebhookDelivery $delivery): JsonResponse
    {
        abort_if(
            $delivery->status === 'delivered',
            422,
            'Delivery already succeeded'
        );

        $delivery->update([
            'status'          => 'pending',
            'attempt_count'   => 0,
            'next_attempt_at' => now(),
        ]);

        SendWebhookJob::dispatch($delivery);

        return response()->json(['queued' => true]);
    }

    // Resend all failed deliveries for a subscription
    public function resendFailed(WebhookSubscription $subscription): JsonResponse
    {
        $count = WebhookDelivery::where('subscription_id', $subscription->id)
            ->where('status', 'failed')
            ->count();

        WebhookDelivery::where('subscription_id', $subscription->id)
            ->where('status', 'failed')
            ->update([
                'status'          => 'pending',
                'attempt_count'   => 0,
                'next_attempt_at' => now(),
            ]);

        WebhookDelivery::where('subscription_id', $subscription->id)
            ->where('status', 'pending')
            ->each(fn($d) => SendWebhookJob::dispatch($d));

        return response()->json(['requeued' => $count]);
    }

    // Attempt history for specific delivery
    public function attempts(WebhookDelivery $delivery): JsonResponse
    {
        return response()->json(
            $delivery->attempts()
                ->orderBy('attempt_number')
                ->get(['attempt_number', 'started_at', 'duration_ms',
                       'response_code', 'response_body', 'error_message', 'success'])
        );
    }
}

Delivery Dashboard

For operational monitoring, you need an interface with filtering by:

• Status (pending, delivered, failed)
• Event type
• Subscription / consumer
• Time range

Useful aggregates in PostgreSQL:

-- Statistics for last 24 hours
SELECT
  event_type,
  COUNT(*) FILTER (WHERE status = 'delivered') AS delivered,
  COUNT(*) FILTER (WHERE status = 'failed')    AS failed,
  COUNT(*) FILTER (WHERE status = 'pending')   AS pending,
  ROUND(AVG(attempt_count), 2)                 AS avg_attempts,
  PERCENTILE_CONT(0.95) WITHIN GROUP (
    ORDER BY EXTRACT(EPOCH FROM (delivered_at - created_at))
  ) AS p95_delivery_seconds
FROM webhook_deliveries
WHERE created_at > NOW() - INTERVAL '24 hours'
GROUP BY event_type
ORDER BY failed DESC;

Log Retention

Delivery attempts consume space. Rotation strategy:

• Successful attempts—keep 30 days, then delete request body, keep metadata
• Failed—keep 90 days (needed for integration audit)
• Error response body—max 2 KB, don't store binary data

Timelines

Logging system + manual resend: 3–5 days. With dashboard, aggregates, filtering, and retention policy: 1–1.5 weeks.