Monitoring Machinery (Go Task Queue) with Vigilmon: Uptime, Heartbeats & Alerts

Machinery is an async task queue and job queue for Go that supports Redis and AMQP (RabbitMQ) as its message broker. When Machinery is down — because Redis ran out of memory, RabbitMQ lost its connection, or a worker panicked — tasks queue silently and your users never notice until something fails visibly hours later. Vigilmon closes that gap. This tutorial wires Machinery into Vigilmon for uptime monitoring, heartbeat pings, and alert routing in about 20 minutes.

What You'll Build

A /health endpoint that checks the Machinery broker (Redis or AMQP) connectivity
A Vigilmon HTTP monitor pointed at that endpoint
A Vigilmon heartbeat ping sent by each recurring Machinery task
Email and Slack webhook alerts

Prerequisites

Go 1.21+
A Machinery project (github.com/RichardKnop/machinery/v2)
Redis 6+ or RabbitMQ as your broker
A free Vigilmon account

Step 1: Expose a Health Endpoint

Machinery workers don't expose an HTTP server by default — you'll add one alongside your worker process. The health check pings the broker and reports whether the worker is consuming tasks.

For a Redis broker

// internal/health/machinery.go
package health

import (
    "context"
    "encoding/json"
    "net/http"
    "time"

    "github.com/redis/go-redis/v9"
)

type MachineryHealthHandler struct {
    Redis *redis.Client
}

type HealthResponse struct {
    Status    string            `json:"status"`
    Timestamp time.Time         `json:"timestamp"`
    Checks    map[string]string `json:"checks"`
}

func (h *MachineryHealthHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
    defer cancel()

    checks := map[string]string{}
    status := "ok"

    if err := h.Redis.Ping(ctx).Err(); err != nil {
        checks["redis"] = "error: " + err.Error()
        status = "degraded"
    } else {
        checks["redis"] = "ok"
    }

    code := http.StatusOK
    if status != "ok" {
        code = http.StatusServiceUnavailable
    }

    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(code)
    json.NewEncoder(w).Encode(HealthResponse{
        Status:    status,
        Timestamp: time.Now().UTC(),
        Checks:    checks,
    })
}

For an AMQP (RabbitMQ) broker

Replace the Redis ping with an AMQP dial check:

import (
    "github.com/rabbitmq/amqp091-go"
)

func checkAMQP(dsn string) error {
    conn, err := amqp091.DialConfig(dsn, amqp091.Config{
        Dial: amqp091.DefaultDial(2 * time.Second),
    })
    if err != nil {
        return err
    }
    conn.Close()
    return nil
}

mux := http.NewServeMux()
mux.Handle("/health", &health.MachineryHealthHandler{Redis: redisClient})

go http.ListenAndServe(":8080", mux)

Test it:

curl -s localhost:8080/health | jq
# {
#   "status": "ok",
#   "timestamp": "2025-06-29T10:00:00Z",
#   "checks": { "redis": "ok" }
# }

Kill Redis and re-run — you'll see 503 with "redis": "error: ...". Vigilmon treats any non-2xx as a failure.

Step 2: Create a Vigilmon HTTP Monitor

| Field | Value | |---|---| | URL | https://workers.yourdomain.com/health | | Method | GET | | Check interval | 60 seconds | | Expected status | 200 | | Timeout | 10 seconds | | Regions | Select 2–3 for triangulation |

Under Alert Channels, add your email address. You'll add Slack in Step 5.

Click Save — Vigilmon begins polling immediately.

Step 3: Heartbeat Monitoring for Recurring Tasks

HTTP uptime checks verify the broker is reachable, but they won't detect a cron task that silently stopped running. The heartbeat pattern fixes this: your Machinery task pings Vigilmon every time it completes successfully. Silence beyond the heartbeat window triggers an alert.

Get your Heartbeat URL from Vigilmon (Dashboard → Heartbeat Monitors → New):

https://vigilmon.online/api/heartbeats/YOUR-UUID/ping

Set the period to 1.5× your task's schedule interval — for a task that runs every 5 minutes, set the period to 7 or 8 minutes.

Define a Machinery task that pings Vigilmon on completion:

// internal/tasks/daily_sync.go
package tasks

import (
    "log"
    "net/http"
    "os"
    "time"
)

var httpClient = &http.Client{Timeout: 5 * time.Second}

// DailySync is a recurring Machinery task.
// Machinery calls this function when a matching message arrives on the broker.
func DailySync() error {
    // --- actual task logic here ---
    if err := syncData(); err != nil {
        return err // returning an error stops the heartbeat ping
    }

    // Ping Vigilmon only after successful completion
    heartbeatURL := os.Getenv("VIGILMON_HEARTBEAT_URL")
    resp, err := httpClient.Get(heartbeatURL)
    if err != nil {
        log.Printf("[heartbeat] ping failed: %v", err)
        return nil // don't fail the task over a missed ping
    }
    resp.Body.Close()
    log.Printf("[heartbeat] pinged, status %d", resp.StatusCode)
    return nil
}

func syncData() error {
    // placeholder
    return nil
}

// main.go (worker side)
server.RegisterTasks(map[string]interface{}{
    "daily_sync": tasks.DailySync,
})

// Sender side — sends the task message on a schedule
ticker := time.NewTicker(5 * time.Minute)
for range ticker.C {
    _, err := server.SendTask(&machinery.Signature{
        Name: "daily_sync",
    })
    if err != nil {
        log.Printf("failed to enqueue daily_sync: %v", err)
    }
}

Store the heartbeat URL in the environment:

VIGILMON_HEARTBEAT_URL=https://vigilmon.online/api/heartbeats/YOUR-UUID/ping

Step 4: Monitor Queue Depth (Advanced)

A long queue depth means workers can't keep up — tasks are piling up but not executing. For Redis-backed Machinery, check queue depth via the Redis list length:

// Add to health handler
queueLen, err := h.Redis.LLen(ctx, "machinery_tasks").Result()
if err != nil {
    checks["queue_depth"] = "unknown"
} else if queueLen > 1000 {
    checks["queue_depth"] = fmt.Sprintf("warning: %d pending tasks", queueLen)
    status = "degraded"
} else {
    checks["queue_depth"] = fmt.Sprintf("ok (%d pending)", queueLen)
}

This surfaces backlog pressure in your Vigilmon dashboard and alerts before tasks start timing out on the consumer side.

Step 5: Alert Routing

Email Alerts

Already configured in Step 2. Vigilmon alerts when:

The /health endpoint returns non-2xx (broker unreachable)
The endpoint times out
The heartbeat window expires (recurring task stopped)

Slack Webhook Alerts

Create a Slack incoming webhook (Slack docs)
In Vigilmon → Alert Channels → Add Channel → Webhook
Paste the Slack webhook URL
Assign to your Machinery monitors

Alert payload:

{
  "text": "🔴 *workers.yourdomain.com/health* is DOWN\nStatus: 503 | Region: eu-west-1\nDuration: 4m 10s"
}

Step 6: Test the Full Loop

Simulate Redis failure: stop Redis and curl /health — expect 503.
Verify Vigilmon detects it: within one check interval the dashboard goes red and an alert fires.
Test heartbeat silence: stop enqueueing the periodic task and wait for the heartbeat window to expire — you receive an alert.
Recover: restart Redis and re-enable the task sender. Vigilmon auto-recovers and sends a "back online" notification.

Production Checklist

[ ] /health checks broker connectivity (Redis PING or AMQP dial)
[ ] Queue depth check included for backlog visibility
[ ] Heartbeat URL stored in environment variable
[ ] Heartbeat period set to 1.5× task schedule interval
[ ] Heartbeat ping sent only on successful task completion
[ ] Alert channels tested end-to-end

Wrapping Up

You now have production-grade monitoring for your Machinery task queue:

Uptime monitoring that catches broker failures within 60 seconds
Queue depth visibility to catch worker backlog before tasks time out
Heartbeat monitoring that catches silent task-scheduling failures

If you hit issues or have questions, drop a comment below. Happy monitoring!