Monitoring Dkron with Vigilmon

Dkron is an open-source distributed cron job scheduler written in Go. It gives you a web UI, a gRPC API, and multi-node clustering via Serf for managing scheduled tasks across your infrastructure — no single point of failure, built-in leader election, and execution history. But Dkron itself has no external health monitoring: if the web server stops accepting connections, the gRPC API hangs, or the Serf cluster loses quorum, your jobs silently stop running. Vigilmon fills that gap, giving you uptime monitors for every Dkron service layer so a job scheduling outage wakes you up before it causes downstream damage.

What You'll Set Up

• HTTP uptime monitor for the Dkron web UI and REST API (port 8080)
• TCP monitor for the gRPC API (port 8946)
• TCP monitor for Serf cluster membership (port 6868)
• Cron heartbeat for job execution success verification
• Monitors for etcd consensus store connectivity
• Leader election and job stale-detection alerting

Prerequisites

• Dkron 3.x installed and running (single-node or cluster)
• Ports 8080 (web/API), 8946 (gRPC), and 6868 (Serf) accessible from your monitoring probe
• A free Vigilmon account

Step 1: Monitor the Dkron Web Server and REST API

Dkron exposes its web UI and REST API on port 8080. This is the primary interface for operators and the integration point for most tooling.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the Dkron health endpoint: http://your-dkron-host:8080/healthz
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Dkron's /healthz endpoint returns 200 OK when the node is healthy. If the process crashes or the HTTP listener stops, this check fails immediately.

For API-level verification, also add a monitor for the jobs list endpoint:

http://your-dkron-host:8080/v1/jobs

Set Expected HTTP status to 200 and Expected response body contains to [ (a JSON array open bracket) — this confirms the API is responding and returning data, not just accepting connections.

Step 2: Monitor the gRPC API Port

Dkron's gRPC API on port 8946 handles inter-node communication and programmatic job scheduling. Add a TCP monitor so you detect failures even when HTTP is still up:

1. Click Add Monitor → TCP Port.
2. Enter Host: your-dkron-host and Port: 8946.
3. Set Check interval to 1 minute.
4. Click Save.

A TCP monitor confirms the gRPC listener is accepting connections. This catches situations where the gRPC server crashes independently of the HTTP layer — common after configuration changes that affect only the gRPC bind address.

Step 3: Monitor Serf Cluster Membership

Dkron uses Serf for cluster membership and gossip-based node discovery. Serf listens on port 6868. In a single-node setup this port is still active; in a cluster, losing Serf connectivity means nodes can't coordinate.

1. Click Add Monitor → TCP Port.
2. Enter Host: your-dkron-host and Port: 6868.
3. Set Check interval to 1 minute.
4. Click Save.

Add one TCP monitor per cluster node. If Serf on any node stops responding, that node is effectively isolated from the cluster and its jobs may not run or may run without deduplication.

To check cluster member count via the REST API, add a second HTTP monitor:

http://your-dkron-host:8080/v1/members

Set Expected response body contains to the number of expected members, e.g. "Status":1 to verify at least one active member is reporting healthy status.

Step 4: Verify Job Execution with Cron Heartbeats

The most important thing to monitor in a job scheduler isn't the server — it's whether jobs are actually running. Use Vigilmon's cron heartbeat to detect silent job failures.

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set Name to Dkron - [job name] execution.
3. Set Expected ping interval to match your job's schedule (e.g. 60 minutes for an hourly job).
4. Set Grace period to 5 minutes.
5. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

In your Dkron job definition, add a shell command that pings Vigilmon after a successful run:

{
  "name": "process-queue",
  "schedule": "@every 1h",
  "command": "bash -c 'python /opt/app/process_queue.py && curl -s https://vigilmon.online/heartbeat/abc123'",
  "owner": "ops@example.com",
  "owner_email": "ops@example.com"
}

The && ensures the heartbeat only fires when the command exits successfully. If the job fails or never runs, Vigilmon alerts after the expected interval plus grace period.

Create one heartbeat monitor per critical job. For jobs that must never be missed (payment processing, backup jobs, certificate renewal), set the grace period to 0 minutes.

Step 5: Monitor etcd Consensus Store Connectivity

Dkron stores job definitions, execution history, and leader election state in etcd. If etcd becomes unreachable, Dkron can't persist new jobs or record execution results.

etcd's client API listens on port 2379 by default. Add a TCP monitor:

1. Click Add Monitor → TCP Port.
2. Enter Host: your-etcd-host and Port: 2379.
3. Set Check interval to 1 minute.
4. Click Save.

For an HTTP-based etcd health check:

http://your-etcd-host:2379/health

Set Expected HTTP status to 200 and Expected response body contains to "health":"true". This confirms etcd is not just listening but is a healthy cluster member.

In a clustered Dkron setup, add etcd monitors for each etcd node. Losing a minority of etcd nodes degrades durability; losing a majority halts all writes and causes Dkron leader election to stall.

Step 6: Leader Election and Stale Job Detection

In a Dkron cluster, only the leader node schedules and dispatches jobs. If leader election fails (e.g. all nodes think they're followers), no jobs run.

Check the leader status via the REST API:

http://your-dkron-host:8080/v1/leader

1. Click Add Monitor → HTTP / HTTPS.
2. Enter the URL above.
3. Set Expected HTTP status to 200.
4. Set Expected response body contains to "Name" (a valid leader response includes a node name).
5. Click Save.

For stale job detection, monitor the execution history endpoint for your most critical job:

http://your-dkron-host:8080/v1/executions/your-job-name

Set Expected response body does not contain to "finished_at":"" if your monitoring tool supports negative body matching — an empty finished_at indicates a stuck execution.

Step 7: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, PagerDuty, or a webhook.
2. For the job heartbeat monitors, set Consecutive failures before alert to 1 — a missed heartbeat is always an incident.
3. For TCP and HTTP monitors, set Consecutive failures before alert to 2 to absorb brief network blips.
4. Use Maintenance windows before planned Dkron upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 10}'

# Upgrade Dkron
systemctl stop dkron
apt-get install dkron
systemctl start dkron

# Maintenance window expires automatically

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP health | :8080/healthz | Dkron process crash, HTTP listener failure | | HTTP API | :8080/v1/jobs | API layer failure, empty job registry | | TCP gRPC | :8946 | gRPC server crash, inter-node comm failure | | TCP Serf | :6868 | Cluster membership disruption | | HTTP leader | :8080/v1/leader | Leader election failure | | TCP etcd | :2379 | Consensus store unavailable | | Cron heartbeat | Vigilmon URL | Job not running, job failing silently |

Dkron's distributed architecture means a failure can be partial — the web UI might still load while job scheduling has silently stopped because Serf lost quorum or etcd is unreachable. With Vigilmon watching every layer from the HTTP endpoint down to the TCP ports and job heartbeats, you'll know exactly which component failed and can respond before your scheduled tasks fall behind.