tutorial

How to Monitor TrilioVault for Kubernetes (Trilio) with Vigilmon (HTTP + TCP + Alerts)

TrilioVault for Kubernetes (TVK) is an application-aware backup and recovery platform that captures entire application state — persistent data, Kubernetes me...

TrilioVault for Kubernetes (TVK) is an application-aware backup and recovery platform that captures entire application state — persistent data, Kubernetes metadata, configurations, and custom resources — in a single atomic snapshot. TVK uses BackupPlan and Backup CRDs to define what gets protected and when, pushes snapshot data to S3, NFS, Azure Blob, or GCS targets, and runs a TVK manager deployment alongside a target browser and on-demand restore jobs. When any part of that pipeline fails silently — manager crash, unreachable target storage, hung controller — your backups stop without any Kubernetes-native alert reaching your team.

In this tutorial you'll set up comprehensive uptime monitoring for TrilioVault for Kubernetes using Vigilmon — free tier, no credit card required.

Why TrilioVault for Kubernetes needs external monitoring

TVK spans multiple components that must all be healthy for backups to complete. Kubernetes pod health checks only tell you a container is running, not that backup jobs are succeeding or that target storage is reachable:

  • TVK manager pod crashes during backup — an in-progress Backup resource is left in an intermediate state; no completion event is emitted and the data snapshot is incomplete on the target
  • Target storage becomes unreachable — if the S3 bucket, NFS export, or Azure container is unreachable, TVK logs an error internally but the BackupPlan controller does not surface a failure event to the namespace, causing silent backup gaps
  • BackupPlan controller hangs with no event — the TVK operator can enter a reconciliation loop after a CRD version skew or webhook timeout; BackupPlan resources appear present but no new Backup objects are created
  • Cross-namespace restore fails when restore job can't reach target — a restore job spawned in a different namespace may lack network policy permissions to reach the backup target, causing the restore to fail after the manager has already handed off control
  • TVK webhook server down blocks new BackupPlan creation — TVK registers a validating webhook; if the webhook server is unavailable, kubectl apply of any BackupPlan or Backup manifest is rejected by the API server with a webhook timeout error

External monitoring from multiple geographic regions catches failures spanning the TVK control plane, webhook server, and backup target storage simultaneously.

What you'll need

  • A Kubernetes cluster with TrilioVault for Kubernetes installed (via Helm or OLM operator)
  • An S3, NFS, Azure Blob, or GCS backup target configured and accessible from the cluster
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose TVK health endpoints

Expose the TVK manager health endpoint

The TVK manager deployment exposes a health endpoint on port 8080. Expose it via NodePort so Vigilmon can reach it from outside the cluster:

# tvk-manager-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: tvk-manager-health
  namespace: tvk
spec:
  type: NodePort
  selector:
    app: k8s-triliovault-backend
  ports:
    - name: healthz
      port: 8080
      targetPort: 8080
      nodePort: 30808
      protocol: TCP

Apply it:

kubectl apply -f tvk-manager-health-svc.yaml

Verify the health endpoint responds:

curl http://<node-ip>:30808/healthz
# {"status":"ok"}

Expose the TVK webhook server

The TVK admission webhook runs on port 9443. Expose it via NodePort to monitor its TCP reachability:

# tvk-webhook-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: tvk-webhook-health
  namespace: tvk
spec:
  type: NodePort
  selector:
    app: k8s-triliovault-backend
  ports:
    - name: webhook
      port: 9443
      targetPort: 9443
      nodePort: 30943
      protocol: TCP

kubectl apply -f tvk-webhook-health-svc.yaml

Expose the TVK target browser

The target browser service lets you verify connectivity to configured backup targets. Expose it on port 8081:

# tvk-target-browser-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: tvk-target-browser-external
  namespace: tvk
spec:
  type: NodePort
  selector:
    app: k8s-triliovault-target-browser
  ports:
    - name: http
      port: 8081
      targetPort: 8081
      nodePort: 30881
      protocol: TCP

kubectl apply -f tvk-target-browser-svc.yaml
curl http://<node-ip>:30881/healthz
# {"status":"ok"}

Confirm all TVK pods are running before proceeding:

kubectl get pods -n tvk
# NAME                                          READY   STATUS    RESTARTS   AGE
# k8s-triliovault-backend-...                   1/1     Running   0          2d
# k8s-triliovault-target-browser-...            1/1     Running   0          2d
# k8s-triliovault-admission-webhook-...         1/1     Running   0          2d

Step 2: Set up HTTP monitoring in Vigilmon

With health endpoints exposed, add TVK to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to http://<node-ip>:30808/healthz
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: ok
  6. Save the monitor as "TVK Manager /healthz"
  7. Repeat steps 1–6 for http://<node-ip>:30881/healthz (target browser), saving it as "TVK Target Browser /healthz"

The multi-region consensus advantage

Vigilmon probes your endpoints from multiple geographic regions simultaneously. Because TVK health depends on both the Kubernetes control plane and external backup target storage, a single probe failure could reflect a transient network path issue rather than a real TVK outage. Vigilmon's multi-region consensus requires multiple independent probes to agree before triggering an alert — giving you accurate detection without false positives during transient blips.

Step 3: Monitor your TCP layer

TCP monitoring catches TVK component failures and backup target reachability problems before they show up as backup failures or HTTP errors.

In Vigilmon:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your node IP and port 30808 (TVK manager health)
  4. Save the monitor as "TVK Manager TCP"
  5. Add a TCP monitor for port 30943 (TVK webhook server) — this is your early warning that BackupPlan creation will be rejected by the API server
  6. Add a TCP monitor for <s3-endpoint-hostname> on port 443 (HTTPS to your S3/Azure/GCS backup target)
  7. If using NFS as a backup target, add a TCP monitor for <nfs-server-ip> on port 2049

Verify target reachability from within the cluster before setting up Vigilmon monitors:

# Test S3 target reachability from a pod
kubectl run tvk-debug --image=curlimages/curl --rm -it --restart=Never -- \
  curl -o /dev/null -sw "%{http_code}" https://<s3-bucket>.s3.<region>.amazonaws.com

# Test NFS target reachability from a node
nc -zv <nfs-server-ip> 2049
# Connection to <nfs-server-ip> 2049 port [tcp/nfs] succeeded!

# Check TVK webhook registration is active
kubectl get validatingwebhookconfigurations | grep triliovault

Step 4: Configure alert channels

A crashed TVK manager leaves backup jobs orphaned with no completion event. An unreachable backup target silently skips every scheduled backup. You need to know immediately.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your on-call or backup-team email address
  3. Assign the channel to your TVK manager, webhook, and target storage monitors

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL (Slack incoming webhook, PagerDuty Events API, etc.)
  3. Assign the channel to your monitors

Example payload Vigilmon sends when a monitor goes down:

{
  "monitor_name": "TVK Manager /healthz",
  "status": "down",
  "url": "http://203.0.113.50:30808/healthz",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 60
}

When you receive a TVK alert, correlate it with the cluster:

# Check TVK manager and related pods
kubectl get pods -n tvk -o wide

# Inspect TVK manager logs for backup or target errors
kubectl logs -n tvk deploy/k8s-triliovault-backend --tail=50

# Check all BackupPlan resources and their last reconcile time
kubectl get backupplan -A

# Check Backup resources for failed or stalled jobs
kubectl get backup -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATUS:.status.status,STARTED:.status.startTimestamp'

# Look for restore jobs that may be stuck
kubectl get restore -A

# Verify the TVK webhook is functioning (test with a dry-run)
kubectl apply --dry-run=server -f <your-backupplan.yaml>

Step 5: Create a public status page

Backup infrastructure health is relevant to multiple teams — storage, platform, and application owners. A public status page lets everyone verify that TVK and its targets are operational without needing kubectl access.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "TrilioVault Backup Infrastructure"
  3. Add your monitors: TVK manager health, TVK webhook TCP, target browser health, S3 target TCP, NFS target TCP
  4. Group them: TVK Manager, TVK Webhook, Backup Target Storage
  5. Publish the page

Share the status page URL in your runbook, with application teams whose data is protected by TVK, and with anyone who needs visibility into backup health without cluster access.

Putting it all together

Here's a summary of the monitors to create for a TrilioVault for Kubernetes deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://<node-ip>:30808/healthz | HTTP | TVK manager health, crash loops, OOMKill during backup | | http://<node-ip>:30881/healthz | HTTP | Target browser health, target connectivity validation | | <node-ip>:30943 | TCP | TVK webhook server reachability — catches BackupPlan creation failures | | <s3-endpoint>:443 | TCP | S3/Azure/GCS backup target reachability — silent backup failures | | <nfs-server-ip>:2049 | TCP | NFS backup target reachability — NFS mount failures on backup jobs |

Verify the full TVK state in your cluster:

# List all TVK CRDs to confirm installation is complete
kubectl get crds | grep trilio

# Summarize BackupPlan status across all namespaces
kubectl get backupplan -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,TARGET:.spec.backupConfig.target.name,SCHEDULE:.spec.backupConfig.scheduleType'

# Check the most recent Backup objects cluster-wide
kubectl get backup -A --sort-by='.metadata.creationTimestamp' | tail -20

# Confirm TVK target resources are healthy
kubectl get target -A

# Inspect the TVK operator logs for reconciliation errors
kubectl logs -n tvk -l app=k8s-triliovault-backend --tail=100 | grep -i error

What's next

  • Multi-target coverage — if you have multiple Target resources (one S3, one NFS, one Azure Blob), add a TCP monitor for each target endpoint so you know exactly which backend is degraded when a backup fails
  • Heartbeat monitors on backup completion — configure your BackupPlan post-hook or a CronJob wrapper to ping a Vigilmon heartbeat URL after each successful backup; if the heartbeat stops arriving, Vigilmon alerts you that scheduled backups are no longer completing
  • Restore test job monitoring — add HTTP monitors for restore validation jobs that run on a schedule; external monitoring confirms that restore jobs are completing successfully, not just that the TVK manager pod is alive

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles