Most applications need some form of background work: send an email after a signup, generate a report, retry a webhook, or delay a job until some future time. Pushing work into a queue is the easy part. The hard part is everything that follows: retries, delayed execution, prioritization, visibility, and recovery when a worker dies in the middle of processing.

The code in task-queue-go builds those pieces on top of Redis, using Redis Streams for delivery and a sorted set for deferred execution. That choice matters because many teams already run Redis in production. If Redis is already in your stack, it is worth understanding how far you can push it before reaching for a larger message bus or a dedicated job system.

The strongest idea in this design is simple: delayed jobs and automatic retries share one deferred-work path. A user-scheduled task and a retrying failed task both sit in tasks:scheduled until they are due. That keeps the implementation smaller, observability more consistent, and the control flow easier to reason about.

This post walks through the architecture, the Redis data model, the worker lifecycle, the retry path, and the queue stats model. The goal is not to sell Redis Streams as a universal answer. It is to show how they can be used to build a practical distributed task queue in Go with explicit tradeoffs.

🏗️ Architecture Overview

The system is split into two binaries. cmd/api-server/ accepts task submissions, exposes admin endpoints, runs the scheduler, and publishes live events. cmd/worker/ joins a Redis consumer group, executes task handlers, applies retry policy, and recovers orphaned work from dead consumers.

Redis does most of the coordination work. Priority streams such as tasks:critical and tasks:normal carry queue delivery. The sorted set tasks:scheduled stores deferred work, which includes both user-scheduled tasks and retryable failures waiting for backoff to expire. Redis Pub/Sub is a separate path for lifecycle events such as task.submitted, task.started, task.retrying, task.completed, task.failed, and worker state changes.

At a high level, the API server either enqueues a task immediately or stores it for future activation. Workers consume from Redis Streams through a consumer group, update task state in Redis, acknowledge messages only after the task has been handled safely, and push exhausted failures into the dead letter queue. When a retryable task fails, the worker does not immediately re-enqueue it. Instead, it computes a backoff delay and routes the task back through deferred execution.

Architecture overview

Source:

🗄️ Redis Data Model

The Redis data model separates transport from task state. Stream entries are small. They carry only the fields a worker needs to look up the real task record, such as task_id and type. The full task payload, timestamps, attempt counters, retry metadata, and result fields live in a separate key like task:{id}.

That split is one of the most important implementation choices in the project. If large payloads were stored directly in stream messages, every delivery, claim, and acknowledgment path would move more data than necessary. By keeping the stream entry lightweight, the stream behaves more like a delivery index than like the canonical storage layer. Redis Streams handle coordination. The task record handles state.

Priority is implemented with separate streams rather than one shared stream plus a priority field. The project uses tasks:critical, tasks:high, tasks:normal, and tasks:low. That makes consumption logic explicit and lets workers listen across all priorities in a known order.

Deferred work lives in the sorted set tasks:scheduled. The score is a future Unix timestamp. A task sitting in that set may be in scheduled state because a user asked for delayed execution, or it may be in retrying state because a worker scheduled it for exponential backoff after failure. The scheduler does not need separate storage for those cases; it only needs to know that the task is due.

Finally, Redis consumer groups maintain a Pending Entries List, or PEL. That is the record of messages that have been delivered to a worker but not yet acknowledged. The PEL is what makes orphan recovery possible, and it is also why the queue reports pending_unacked separately from queued. Backlog and in-flight work are related, but they are not the same number.

Redis data model

Source:

⚡ Priority Delivery with Redis Streams

Priority is implemented with four separate streams instead of one shared stream with a priority field. A worker does not need to inspect a message, compare priorities, and then decide whether it should keep reading. It can ask Redis for work from all priority streams at once.

In DequeueBlocking, the worker builds a stream list in priority order: critical, high, normal, then low. Redis Streams consumer groups then return the first available message from that read. This gives the system a clear priority bias. Higher-priority streams are checked first, and in practice they win most contention. But it is still more accurate to describe this as priority-biased delivery than as a strict global priority guarantee. Timing across multiple streams and multiple workers makes exact ordering semantics too strong a claim.

The dequeue path is small enough to show directly:

func (q *RedisQueue) DequeueBlocking(ctx context.Context, consumerID string) (*task.Task, string, error) {
	priorities := []task.Priority{
		task.PriorityCritical,
		task.PriorityHigh,
		task.PriorityNormal,
		task.PriorityLow,
	}

	streams := make([]string, 0, len(priorities)*2)
	for _, p := range priorities {
		streams = append(streams, p.StreamName(q.streamPrefix))
	}
	for range priorities {
		streams = append(streams, “>”)
	}

	result, err := q.client.XReadGroup(ctx, &redis.XReadGroupArgs{
		Group:    q.consumerGroup,
		Consumer: consumerID,
		Streams:  streams,
		Count:    1,
		Block:    q.blockTimeout,
	}).Result()

The same section of code also shows why the queue is at-least-once rather than exactly-once. A message stays in the consumer group’s PEL until the worker explicitly acknowledges it. If the worker crashes after receiving the message but before ACKing it, another worker can claim it later. That is the right tradeoff for a job system: durability and recoverability matter more than pretending duplicates can never happen. The cost is that handlers should be safe to run more than once.

Source:

internal/queue/redis_streams.go#L179-L238

🔁 One Scheduler for Delayed Tasks and Retries

The key design choice is that the system does not maintain one path for delayed jobs and another for retries. Both move through the deferred execution path.

When the API receives a task with a future scheduled_at, it stores the task record, sets state to scheduled, and writes the task ID into tasks:scheduled with the scheduled timestamp as the score. Nothing enters a priority stream yet because the task is not ready to run. The scheduler loop is responsible for waking it later.

That entry point lives in internal/api/handlers/task.go:

if req.ScheduledAt != nil && req.ScheduledAt.After(time.Now().UTC()) {
	t.State = task.StateScheduled

	if err := h.scheduleTask(r.Context(), t, *req.ScheduledAt); err != nil {
		logger.Error().Err(err).Str(“task_id”, t.ID).Msg(“failed to schedule task”)
		h.respondError(w, http.StatusInternalServerError, “failed to schedule task”)
		return
	}

	metrics.RecordTaskSubmission(t.Type, t.Priority.String())
	metrics.RecordScheduledTask()
	h.publishTaskEvent(r.Context(), events.EventTaskSubmitted, t, nil)

	h.respondJSON(w, http.StatusCreated, t.ToResponse())
	return
}

The retry path deliberately reuses that design. When a worker executes a task and the handler fails, the worker first records failure on the task, then uses the retry policy to compute a backoff delay. The task transitions to retrying, gets a new scheduled_at, and is written into the same sorted set. Only after that deferred state is safely persisted does the worker acknowledge the old stream message. In other words, retry is not implemented as “put the message back right away.” Retry is implemented as “schedule future work.”

if t.CanRetry() {
	if err := sm.Fail(execErr.Error()); err != nil {
		log.Error().Err(err).Msg(“failed to mark task as failed before retry”)
	}

	retryer := task.NewRetryer(p.retryPolicy)
	if _, err := retryer.ScheduleRetry(t); err != nil {
		log.Error().Err(err).Msg(“failed to schedule retry — falling back to DLQ”)
		_ = p.queue.UpdateTask(ctx, t)
		_ = p.dlq.Add(ctx, t, “retry scheduling failed: ”+err.Error())
		_ = p.queue.Acknowledge(ctx, t, messageID)
		return
	}

	if err := p.queue.UpdateTask(ctx, t); err != nil {
		log.Error().Err(err).Msg(“failed to persist retrying task”)
		return
	}

	if err := p.scheduleTask(ctx, t, *t.ScheduledAt); err != nil {
		log.Error().Err(err).Msg(“failed to add task to scheduled set — leaving in PEL for orphan recovery”)
		return
	}

	if err := p.queue.Acknowledge(ctx, t, messageID); err != nil {
		log.Error().Err(err).Msg(“failed to acknowledge task after scheduling retry”)
	}
	return
}

The scheduler does not care why a task is in tasks:scheduled. It only cares that the score is due and the task state is one of the deferred states it knows how to reactivate. When the wake-up time arrives, the scheduler transitions the task back to pending and pushes a lightweight stream entry into the correct priority stream.

if t.State != task.StateScheduled && t.State != task.StateRetrying {
	s.client.ZRem(ctx, scheduledSetKey, taskID)
	return nil
}

sm := task.NewStateMachine(t)
if err := sm.Transition(task.StatePending); err != nil {
	return fmt.Errorf(“failed to transition task: %w”, err)
}

streamName := t.Priority.StreamName(s.queue.streamPrefix)
_, err = s.client.XAdd(ctx, &redis.XAddArgs{
	Stream: streamName,
	Values: map[string]interface{}{
		“task_id”: t.ID,
		“type”:    t.Type,
	},
}).Result()

This unification has three benefits. It removes duplicated logic. It gives scheduled jobs and retries the same observability model. And it makes the control flow easier to describe: future work is future work, whether a user requested delay explicitly or a retry policy requested it after failure.

Unified scheduler flow

Source:

🔀 Task State Machine

The task model uses an explicit state machine instead of relying on scattered boolean flags or loosely coordinated timestamps. The main states are pending, scheduled, running, completed, failed, retrying, canceled, and dead_letter. That list captures both the happy path and the operational path, which matters because job systems spend a lot of their complexity budget on failure cases rather than on success.

The real value is not the names themselves. The value is the transition map. pending can become running or scheduled. running can become completed, failed, or retrying. retrying can move back to pending when the scheduler wakes it. dead_letter can move back to pending when an operator manually requeues it. Those rules are declared centrally in code.

var ValidTransitions = map[State][]State{
	StatePending:    {StateScheduled, StateRunning, StateCanceled},
	StateScheduled:  {StatePending, StateRunning, StateCanceled},
	StateRunning:    {StateCompleted, StateFailed, StateRetrying, StateCanceled},
	StateRetrying:   {StateRunning, StateFailed, StateDeadLetter, StateCanceled, StatePending},
	StateFailed:     {StateRetrying, StateDeadLetter, StatePending},
	StateCompleted:  {},
	StateCanceled:   {},
	StateDeadLetter: {StatePending},
}

For a distributed queue, that clarity is worth a lot. When the same task can be submitted, delayed, started, retried, recovered, canceled, and dead-lettered, implicit state logic becomes a debugging trap very quickly.

Task state machine

Source:

internal/task/state.go#L87-L96

🛡️ Failure Handling and Reliability

The retry path is deliberately cautious. A worker does not jump straight from a runtime error to immediate re-enqueue. It first marks the task failed, computes backoff with the retry policy, transitions the task into retrying, and sets a future scheduled_at. The API response model exposes both scheduled_at and next_retry_at, so the deferred retry is visible outside the worker.

The backoff itself is straightforward exponential growth with a cap and optional jitter:

func (p *RetryPolicy) CalculateBackoff(attempt int) time.Duration {
	if attempt <= 0 {
		return p.InitialBackoff
	}

	backoff := float64(p.InitialBackoff) * math.Pow(p.BackoffFactor, float64(attempt))
	if backoff > float64(p.MaxBackoff) {
		backoff = float64(p.MaxBackoff)
	}
	if p.JitterFactor > 0 {
		jitter := backoff * p.JitterFactor * (rand.Float64()*2 - 1)
		backoff += jitter
	}
	if backoff < 0 {
		backoff = float64(p.InitialBackoff)
	}

	return time.Duration(backoff)
}

Only after the task is written safely back into deferred storage does the worker acknowledge the original stream message. That ordering matters. The system is trying to guarantee that work is either still claimable from the stream or has already been converted safely into future work.

If a task has exhausted its retry budget, the worker stops scheduling future attempts and moves it to the dead letter queue. That path is intentionally separate from retrying because exhaustion is no longer a temporary execution problem; it is an operator-facing problem. The task remains inspectable and can be requeued manually later if someone decides the failure condition has been resolved.

Consumer groups make orphan recovery possible through the Pending Entries List. If a worker crashes after reading a message but before acknowledging it, the message does not disappear. It remains pending. Another worker can inspect idle pending entries and claim them with XCLAIM after the configured idle timeout. That path is intentionally different from retry backoff. Orphan recovery is crash recovery, not an execution failure, so the recovered task is prepared for immediate re-enqueue.

Taken together, these paths define the reliability model of the queue. It is at-least-once. It recovers abandoned work. It delays retryable failures instead of hammering the same dependency instantly. And it preserves irrecoverable work in a DLQ instead of silently dropping it.

Source:

📊 Truthful Queue Observability

One of the easiest ways to make a queue system confusing is to collapse all work into one number and call it “depth.” This project originally had that problem, and fixing it made the system easier to operate and easier to explain.

The current queue stats model separates four different concepts. queued means stream backlog: the number of entries still present in each priority stream. pending_unacked means the consumer group’s PEL: messages that have already been delivered to a worker but not yet acknowledged. Those are in-flight. scheduled_count means deferred work waiting in the sorted set, which includes both future user-scheduled jobs and tasks in retry backoff. dlq_size means exhausted failures parked in the dead letter queue.

Those numbers answer different operational questions. If queued is rising, producers are outpacing consumers. If pending_unacked is rising, workers may be slow, stuck, or dead. If scheduled_count is rising, the system is holding a lot of future work. If dlq_size is rising, something is failing repeatedly and needs inspection.

The type definitions make that distinction explicit:

type PriorityStats struct {
	Queued         int64 `json:“queued”`
	PendingUnacked int64 `json:“pending_unacked”`
}

type QueueStats struct {
	Queues         map[string]*PriorityStats `json:“queues”`
	ScheduledCount int64                     `json:“scheduled_count”`
	DLQSize        int64                     `json:“dlq_size”`
	Totals         struct {
		Queued         int64 `json:“queued”`
		PendingUnacked int64 `json:“pending_unacked”`
		Deferred       int64 `json:“deferred”`
	} `json:“totals”`
}

func (q *RedisQueue) GetQueueStats(ctx context.Context) (*QueueStats, error) {
	for _, p := range priorities {
		streamName := p.StreamName(q.streamPrefix)
		ps := &PriorityStats{}

		length, err := q.client.XLen(ctx, streamName).Result()
		if err == nil {
			ps.Queued = length
		}

		groups, err := q.client.XInfoGroups(ctx, streamName).Result()
		if err == nil {
			for _, g := range groups {
				if g.Name == q.consumerGroup {
					ps.PendingUnacked = g.Pending
					break
				}
			}
		}
	}

Technical readers notice sloppy queue terminology immediately. Redis Streams do give you useful counters, but you still need to name them honestly. A backlog number is not the same as an in-flight number, and neither is the same as deferred work.

Truthful queue observability

Source:

📡 Live Events Without Polling Everything

The system also publishes task and worker lifecycle events as they happen. The API emits events when tasks are submitted. Workers emit events when tasks start, retry, complete, or fail. Heartbeat and admin paths emit worker join, leave, pause, and resume events. Those events are written to Redis Pub/Sub rather than being inferred by clients through repeated polling.

From there, the WebSocket hub subscribes once and fans messages out to any connected client. That design keeps the event path separate from the task delivery path. Redis Streams handle durable queue delivery. Pub/Sub handles live notifications.

The architectural point is not the dashboard itself. The queue has an explicit event model. Readers who want to inspect that model can run the repository locally and watch the lifecycle stream for themselves. That is more useful than a static screenshot because it lets them see retries, worker changes, and completions in their own environment.

Source:

🚀 Run It Locally

The project is meant to be runnable, not just readable. Start Redis locally, run the API, and run a worker in a separate shell:

make dev-redis
make run-api
make run-worker

Then submit an immediate task:

curl -X POST http://localhost:8080/api/v1/tasks \
  -H 'Content-Type: application/json' \
  -d '{
    “type”: “email”,
    “payload”: {“to”: “user@example.com”, “subject”: “hello”},
    “priority”: 1
  }'

Or submit a delayed task by adding scheduled_at. From there, inspect GET /api/v1/tasks, GET /api/v1/tasks/{id}, the admin endpoints, or the optional local dashboard. That is the right level of proof for this kind of system: runnable code, not a curated screenshot.

⚖️ Tradeoffs and Limits

This design makes deliberate tradeoffs. Delivery is at-least-once, which means handlers should be idempotent or at least duplicate-tolerant. Priority is biased, not mathematically strict. Redis is a strong fit for small and medium systems that want a practical queue without introducing more infrastructure, but it is not automatically the right answer for every throughput target or every retention requirement.

There are also operational tradeoffs. Stream backlog, in-flight work, deferred work, and dead-letter backlog are all different numbers that need to be read separately. That is more honest, but it is less convenient than showing one single metric on a dashboard and pretending it explains everything.

The upside is simplicity. One Redis dependency can handle queue delivery, deferred activation, worker recovery metadata, and live events. If your system already has Redis and your workload fits inside its operational envelope, that is a powerful simplification.

✅ Conclusion

Redis Streams are enough to support serious queueing patterns when the design is explicit about state, retries, and observability. In this project, the biggest architectural win is not the use of Streams by itself. It is the decision to reuse the same deferred-work path for both scheduled jobs and retry backoff.

That decision simplifies the rest of the codebase in useful ways. The state machine stays clearer. The scheduler stays smaller. Queue stats become easier to name honestly. The event model becomes easier to follow.

If you want to inspect the behavior yourself, the repository is the companion to this post. Run Redis, start the API and worker, submit a few tasks, and watch how they move through the system. That is the most useful way to evaluate a task queue design: not by screenshots, but by reading the code and running it.

Repository:

github.com/maumercado/task-queue-go

Hold on! Good things are coming your way.

Thanks for sticking with me!