How to Monitor Metacontroller with Vigilmon

Metacontroller is a lightweight Kubernetes add-on that makes it easy to write custom controllers in any language. Instead of requiring you to implement the full Kubernetes controller machinery, Metacontroller handles the watch-reconcile loop and calls your code via HTTP webhooks — a sync webhook for reconciling resources and a finalize webhook for cleanup on deletion. This lets teams write operators in Python, Node.js, Ruby, or any language that can serve HTTP, without binding Go or controller-runtime knowledge.

But Metacontroller is infrastructure that underpins every controller you build on it. When Metacontroller itself goes down, all of your webhook-based controllers stop reconciling. When your sync webhooks become slow or unavailable, resources drift without any Kubernetes-level indication. In this tutorial you'll set up uptime and response-time monitoring for your Metacontroller deployment and webhook servers using Vigilmon — free tier, no credit card required.

Why Metacontroller needs external monitoring

Metacontroller's architecture — a central controller calling out to your HTTP webhooks — creates multiple independent failure points:

• Metacontroller pod down — the central controller that drives all webhook invocations stops; every CompositeController and DecoratorController you've defined stops reconciling silently
• Sync webhook unreachable — Metacontroller calls your sync webhook but gets a connection refused or timeout; the resource stays in its current state and Metacontroller logs the failure without any cluster-level alert
• Slow webhook causing queue backup — your sync webhook is slow (database query, external API call); Metacontroller's work queue backs up, and recently-created resources take minutes to reach their desired state
• Finalize webhook failing — resources with finalizers can't be deleted because the finalize webhook keeps returning errors; objects accumulate in a terminating state and garbage collection stalls
• Custom Resource schema diverged — a CRD version update makes the resource shape incompatible with your sync webhook logic; the webhook returns 500 errors and Metacontroller retries indefinitely

External monitoring from Vigilmon catches all of these at the HTTP level before users file bugs about resources that won't converge.

What you'll need

• Metacontroller installed in your cluster (kubectl apply -k github.com/metacontroller/metacontroller/manifests/production)
• At least one CompositeController or DecoratorController with a sync webhook
• A free Vigilmon account

Step 1: Expose Metacontroller's health endpoint

Metacontroller exposes a /healthz endpoint on port 9999. Make it reachable externally:

# Check the current Metacontroller service
kubectl get svc -n metacontroller

# Expose the health port externally
kubectl expose deployment metacontroller \
  --name=metacontroller-health \
  --type=LoadBalancer \
  --port=9999 \
  --target-port=9999 \
  --namespace=metacontroller

kubectl get svc metacontroller-health -n metacontroller
# NAME                    TYPE           EXTERNAL-IP     PORT(S)
# metacontroller-health   LoadBalancer   203.0.113.54    9999:32999/TCP

# Verify
curl http://203.0.113.54:9999/healthz
# ok

If you prefer Ingress:

# metacontroller-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: metacontroller-health-ingress
  namespace: metacontroller
spec:
  ingressClassName: nginx
  rules:
    - host: meta.yourdomain.com
      http:
        paths:
          - path: /healthz
            pathType: Exact
            backend:
              service:
                name: metacontroller-health
                port:
                  number: 9999

Step 2: Add a health endpoint to your sync webhook server

Your sync webhook server is just an HTTP server — add a /health route that also validates your webhook's dependencies are reachable:

// webhook-server.js (Node.js example)
const http = require("http");

const PORT = 8080;
let lastSyncTime = null;
let syncErrors = 0;

// Your sync webhook handler
function handleSync(parent, children) {
  // Your reconciliation logic here
  lastSyncTime = new Date().toISOString();
  return {
    status: { phase: "Running", observedGeneration: parent.metadata.generation },
    children: [],
  };
}

const server = http.createServer((req, res) => {
  // Health endpoint for Vigilmon
  if (req.url === "/health" && req.method === "GET") {
    const healthy = syncErrors < 10;
    res.writeHead(healthy ? 200 : 503, { "Content-Type": "application/json" });
    res.end(
      JSON.stringify({
        status: healthy ? "ok" : "degraded",
        last_sync: lastSyncTime,
        recent_errors: syncErrors,
      })
    );
    return;
  }

  // Sync webhook endpoint
  if (req.url === "/sync" && req.method === "POST") {
    let body = "";
    req.on("data", (chunk) => (body += chunk));
    req.on("end", () => {
      try {
        const payload = JSON.parse(body);
        const response = handleSync(payload.parent, payload.children);
        syncErrors = Math.max(0, syncErrors - 1); // cool down on success
        res.writeHead(200, { "Content-Type": "application/json" });
        res.end(JSON.stringify(response));
      } catch (err) {
        syncErrors++;
        console.error("Sync error:", err.message);
        res.writeHead(500, { "Content-Type": "application/json" });
        res.end(JSON.stringify({ error: err.message }));
      }
    });
    return;
  }

  res.writeHead(404);
  res.end();
});

server.listen(PORT, () => console.log(`Webhook server on :${PORT}`));

Expose the webhook server with a Service and Ingress so both Metacontroller (for sync calls) and Vigilmon (for health checks) can reach it:

# webhook-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-webhook-server
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app: my-webhook-server
  ports:
    - name: http
      port: 8080
      targetPort: 8080
      protocol: TCP

kubectl apply -f webhook-service.yaml
kubectl get svc my-webhook-server
# NAME               TYPE           EXTERNAL-IP     PORT(S)
# my-webhook-server  LoadBalancer   203.0.113.55    8080:31080/TCP

curl http://203.0.113.55:8080/health
# {"status":"ok","last_sync":null,"recent_errors":0}

Step 3: Set up HTTP monitoring in Vigilmon

With both Metacontroller and your webhook servers exposing health endpoints, add them to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add a monitor for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | Metacontroller healthz | http://203.0.113.54:9999/healthz | 200 | | Sync webhook health | http://203.0.113.55:8080/health | 200 | | Finalize webhook health | http://203.0.113.55:8080/health | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, set status code 200 and optionally match "status":"ok" for webhook monitors
6. Save each monitor

When Metacontroller is down, Vigilmon detects it within 60 seconds — long before your team would otherwise notice that resources have stopped converging.

Step 4: Monitor response time for sync webhooks

Slow sync webhooks are a silent performance problem — resources eventually converge but much later than expected. Vigilmon's response-time monitoring catches webhook latency regression:

1. On each webhook health monitor, go to Advanced settings
2. Enable Response time alerting
3. Set an alert threshold of 2000 ms (Metacontroller's default sync webhook timeout is typically 10s, so alerting at 2s gives you margin to investigate before timeouts start)
4. Save the configuration

If your sync webhook starts taking 3-4 seconds instead of 50ms, Vigilmon will alert you before Metacontroller begins timing out and retrying — turning a performance regression into a correctness problem.

Step 5: Configure alert channels

A Metacontroller outage is a platform-wide event — every controller built on it stops working simultaneously.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your platform team's on-call email
3. Assign the channel to Metacontroller and webhook monitors

Webhook alerts for incident management

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your PagerDuty, Opsgenie, or Slack webhook URL
3. Vigilmon sends this payload on failure:

{
  "monitor_name": "Metacontroller healthz",
  "status": "down",
  "url": "http://203.0.113.54:9999/healthz",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 120
}

Wire the Metacontroller alert into a high-priority incident channel — this alert means all webhook-based controllers across your cluster are offline simultaneously.

Step 6: Correlate Vigilmon alerts with Metacontroller diagnostics

When you get an alert from Vigilmon, run these checks:

# 1. Check Metacontroller pod status
kubectl get pods -n metacontroller

# 2. Read Metacontroller logs for controller errors and webhook failures
kubectl logs -n metacontroller deployment/metacontroller --tail=100

# 3. List your CompositeControllers and DecoratorControllers
kubectl get compositecontroller
kubectl get decoratorcontroller

# 4. Check the sync status of managed resources
# (Replace myapp with your managed resource type)
kubectl get myapp -A \
  -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}'

# 5. Check for terminating resources (finalize webhook failures)
kubectl get myapp -A \
  -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.metadata.deletionTimestamp}{"\n"}{end}' \
  | grep -v "	$"

# 6. Directly probe your sync webhook to confirm it's responding
curl -X POST http://203.0.113.55:8080/sync \
  -H "Content-Type: application/json" \
  -d '{"parent":{"apiVersion":"v1","kind":"Test","metadata":{"name":"test"},"spec":{}},"children":{}}'

If Metacontroller is up but resources aren't converging, the problem is likely the sync webhook — check whether the webhook server pod is running and whether it can reach its dependencies (databases, APIs).

Step 7: Create a status page for webhook-based controllers

Teams that depend on resources managed by your Metacontroller-based controllers benefit from a shared status page:

1. Go to Status Pages → New Status Page
2. Name it: "Platform Controller Status"
3. Add your monitors: Metacontroller healthz, Sync webhook health
4. Share the URL with all teams whose custom resources depend on these controllers

When resources stop converging, teams can check the status page before escalating to the platform team — saving the "why is my resource stuck?" investigation round-trip.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on Metacontroller /healthz | Metacontroller pod failures — all webhook controllers stop | | HTTP monitor on sync webhook /health | Webhook server crashes, dependency failures, high error rates | | Response time alerting on webhook | Latency regression before Metacontroller timeouts begin | | TCP monitor on webhook port | Port-level outages, container crashes before HTTP fails | | Email + webhook alert channels | Immediate notification on platform-wide controller failure | | Status page | Team self-service for stuck custom resource diagnosis |

Metacontroller's simplicity is its strength — any language, any team, any reconciler. But that simplicity means you must supply your own observability. External monitoring from Vigilmon closes that gap, giving you a real-time view of both the Metacontroller core and every webhook server that extends it.

Get started at vigilmon.online — free tier, no credit card required.