tutorial

How to Monitor External-DNS with Vigilmon (Controller Health, DNS Sync, Provider Connectivity)

External-DNS automates one of the most error-prone parts of Kubernetes operations: keeping DNS records synchronized with your cluster's services and ingresse...

External-DNS automates one of the most error-prone parts of Kubernetes operations: keeping DNS records synchronized with your cluster's services and ingresses. When you create a Service of type LoadBalancer or an Ingress resource with a hostname, External-DNS reads it and creates or updates the corresponding DNS records in your configured provider — Route53, Cloudflare, Azure DNS, Google Cloud DNS, and many others.

The problem is that External-DNS is easy to break and easy to ignore. When it stops working, no new services get DNS entries, but existing services continue working fine. Teams don't notice until they deploy a new service and discover DNS records never appeared — often after hours of debugging application configuration that was never the real issue.

In this tutorial you'll set up monitoring for External-DNS using Vigilmon to catch controller failures and sync problems as soon as they occur.

Why External-DNS needs external monitoring

External-DNS failures are insidious because they don't affect running services. Only new services, ingresses, and record updates are broken. Here are the failure modes that teams regularly miss:

  • Controller crashes — the External-DNS pod fails and restarts or enters CrashLoopBackOff. All DNS automation stops. No records are created or updated until the pod recovers.
  • Provider API connectivity lost — the controller is running but can't reach your DNS provider's API (Route53, Cloudflare, etc.) due to a network policy change, expired credentials, or an API rate limit. The controller keeps running but no records are synced.
  • Credentials expired — the IAM role, service account, or API token used by External-DNS expires. The controller logs errors but stays running. DNS records become stale as services change.
  • Reconciliation loop stalls — the controller's reconciliation loop stops making progress, perhaps due to a large batch of records, a deadlock in provider interaction, or a memory issue. Records diverge from the desired state silently.
  • Webhook validation failure — if you run External-DNS with a webhook provider, a failure in the webhook receiver breaks all DNS operations.

The common thread: External-DNS can be completely broken while appearing healthy from a kubectl get pods perspective.

What you'll need

  • A Kubernetes cluster running External-DNS (v0.13+ recommended)
  • External-DNS configured with a supported DNS provider
  • Access to the health/metrics port (default 7979)
  • A free Vigilmon account

Step 1: Locate the External-DNS health endpoint

External-DNS exposes an HTTP health endpoint and Prometheus metrics. The default port is 7979:

# Find the External-DNS deployment
kubectl get deployment -A | grep external-dns

# Check what port it's using
kubectl describe deployment external-dns -n external-dns | grep -A5 "Ports"

Test the health endpoint from inside the cluster:

# Port-forward to test
kubectl port-forward -n external-dns deployment/external-dns 7979:7979 &

# Test the health endpoint
curl http://localhost:7979/healthz
# Response: OK

# Check metrics
curl http://localhost:7979/metrics | grep external_dns
kill %1

The /healthz endpoint returns OK when the controller is running. The /metrics endpoint exposes reconciliation counters that indicate sync health.

Step 2: Expose the health endpoint for external monitoring

To monitor External-DNS with Vigilmon, you need to make the health endpoint reachable from outside the cluster. The cleanest approach is a NodePort service:

# external-dns-health-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: external-dns-health
  namespace: external-dns
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: external-dns
  ports:
    - name: health
      port: 7979
      targetPort: 7979
      nodePort: 30979
      protocol: TCP

kubectl apply -f external-dns-health-nodeport.yaml

Verify it works from outside the cluster:

curl http://<any-node-ip>:30979/healthz
# Response: OK

Step 3: Set up controller health monitoring in Vigilmon

Add the health check as a Vigilmon monitor:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: http://<node-ip>:30979/healthz
  4. Check interval: 1 minute
  5. Under Expected response:
    • Status code: 200
    • Response body contains: OK
  6. Save as "External-DNS Controller Health"

This confirms the controller process is alive. A failed check means the pod has crashed, been evicted, or entered a state where it can't serve health checks.

Step 4: Monitor DNS record sync status

The health endpoint tells you the controller is running, but not whether it's successfully syncing records to your DNS provider. To monitor sync health, expose the metrics endpoint:

# external-dns-metrics-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: external-dns-metrics
  namespace: external-dns
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: external-dns
  ports:
    - name: metrics
      port: 7979
      targetPort: 7979
      nodePort: 30979
      protocol: TCP

The key metrics for sync health are:

# Successful sync operations
external_dns_registry_records{registry="..."} <count>

# Source records read from Kubernetes
external_dns_source_endpoints_total <count>

# Errors in sync operations
external_dns_errors_total <count>

For Vigilmon monitoring, use the metrics endpoint as a health signal — if it stops responding, the controller is unhealthy. Add a second monitor:

  1. Go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: http://<node-ip>:30979/metrics
  4. Check interval: 2 minutes
  5. Expected status code: 200
  6. Response body contains: external_dns
  7. Save as "External-DNS Metrics Endpoint"

Step 5: Monitor provider API connectivity

External-DNS must reach your DNS provider's API to create records. The most reliable way to monitor this connection from outside the cluster is to check that records you expect to exist actually resolve correctly in DNS.

For example, if External-DNS should create a record api.yourdomain.com pointing to your ingress IP:

# Verify the record resolves correctly
dig +short api.yourdomain.com
# Expected: <your-ingress-ip>

Set up a Vigilmon HTTP monitor for each critical DNS-managed endpoint:

  1. Go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://api.yourdomain.com/health
  4. Check interval: 1 minute
  5. Expected status code: 200
  6. Save as "DNS Sync Check — api.yourdomain.com"

If External-DNS stops syncing and an ingress IP changes, this monitor will catch the mismatch — the old DNS record will point to a defunct IP or the new IP's health check will fail with a cert mismatch.

Step 6: Monitor the reconciliation loop

External-DNS runs a reconciliation loop on a configured interval (default 1 minute with --interval=1m). If the loop stalls, records stop being updated even if the controller appears healthy.

Create a test service specifically for monitoring reconciliation — an endpoint that changes predictably so you can verify External-DNS is actively syncing:

# external-dns-canary-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: dns-health-canary
  namespace: external-dns
  annotations:
    external-dns.alpha.kubernetes.io/hostname: dns-canary.yourdomain.com
    external-dns.alpha.kubernetes.io/ttl: "30"
spec:
  type: LoadBalancer
  selector:
    app: dns-health-canary
  ports:
    - port: 80
      targetPort: 8080

Then monitor the canary DNS name with Vigilmon:

  1. Go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: http://dns-canary.yourdomain.com
  4. Check interval: 5 minutes
  5. Save as "External-DNS Reconciliation Canary"

If this monitor fails, External-DNS has stopped writing records to your DNS provider.

Step 7: Configure alert channels

External-DNS failures have a delayed blast radius — they break new deployments rather than existing ones. Configure alerts so the right people are notified without triggering P1 pagers for every transient failure:

  1. Go to Alert Channels → Add Channel → Email
  2. Add your platform engineering team email
  3. Go to Alert Channels → Add Channel → Webhook
  4. Add your Slack webhook URL for #platform-alerts

For the controller health monitor: route to Slack and email. A crashed controller should be fixed within an hour before it blocks deployments.

For the metrics endpoint: route to Slack only. The metrics endpoint going down temporarily is less urgent than the health endpoint.

For DNS sync canary: route to email and Slack. A failed canary means DNS is actively broken for new deployments.

Set confirmation thresholds:

  • Controller health: 2 consecutive failures (accounts for pod restarts)
  • Metrics endpoint: 3 consecutive failures
  • DNS sync canary: 2 consecutive failures

Correlating Vigilmon alerts with External-DNS state

When an External-DNS alert fires:

# 1. Check pod status and restarts
kubectl get pods -n external-dns -o wide

# 2. Check recent logs — look for provider errors
kubectl logs -n external-dns deployment/external-dns --since=10m | \
  grep -E "error|Error|failed|Failed"

# 3. Check for credential or API errors
kubectl logs -n external-dns deployment/external-dns --since=1h | \
  grep -iE "unauthorized|forbidden|rate limit|throttl"

# 4. Check sync events
kubectl get events -n external-dns --sort-by=.lastTimestamp | tail -20

# 5. Verify the DNS provider credentials secret
kubectl get secret -n external-dns
kubectl describe secret <credentials-secret> -n external-dns

Provider-specific checks:

# For Route53: verify IAM role or credentials
aws sts get-caller-identity

# For Cloudflare: test the API token
curl -H "Authorization: Bearer <token>" \
  "https://api.cloudflare.com/client/v4/user/tokens/verify"

# For Azure DNS: check service principal
az account show

Monitoring summary

| Monitor | Type | Interval | What it catches | |---|---|---|---| | External-DNS Controller Health | HTTP | 60s | Controller crash, pod eviction | | External-DNS Metrics Endpoint | HTTP | 2min | Process health, metrics availability | | DNS Sync Canary | HTTP | 5min | Reconciliation loop stall, provider API failure | | api.yourdomain.com | HTTP | 60s | Stale records after IP change |

What's next

  • SSL certificate monitoring — External-DNS often pairs with cert-manager. Vigilmon monitors your certificate expiry and alerts you before a rotation failure leaves users with TLS errors.
  • Multi-provider monitoring — if you use External-DNS with multiple providers in split configurations, create canary records for each provider and monitor them independently.
  • Status page — expose a Vigilmon status page for your cluster's DNS health so application teams can check if External-DNS is working without needing cluster access.

Start monitoring External-DNS for free at vigilmon.online — catch DNS sync failures before they block your next deployment.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles