How to Monitor JuiceFS with Vigilmon (HTTP, TCP, and Heartbeat Monitoring)

JuiceFS is a distributed POSIX-compatible file system built on top of object storage and a metadata engine (Redis, TiKV, or PostgreSQL). It's widely used for shared storage in Kubernetes clusters, AI/ML workloads, and data pipelines. When JuiceFS is healthy, reads and writes are transparent to applications. When it breaks — metadata engine down, object storage unreachable, or CSI driver failure — every workload that mounts the filesystem blocks or crashes.

In this tutorial you'll set up end-to-end monitoring for your JuiceFS infrastructure using Vigilmon — free tier, no credit card required.

Why JuiceFS needs dedicated monitoring

JuiceFS has a multi-layer architecture, and each layer can fail independently:

• Metadata engine failure — if Redis, TiKV, or PostgreSQL (whichever is your metadata store) becomes unreachable, all file operations block immediately; no new opens, no stats, no directory listings
• Object storage outage — the data layer (S3, OSS, MinIO, Ceph) becomes unavailable; reads that hit a cache miss block indefinitely; writes fail silently or return EIO
• JuiceFS CSI driver crash — the juicefs-csi-controller or juicefs-csi-node pod crashes; new pods that request a JuiceFS volume get stuck in Pending or ContainerCreating forever
• Mount process hang — a juicefs mount process or FUSE kernel module enters a hung state; processes accessing the mountpoint block in uninterruptible sleep (D state) until the mount is forcibly unmounted
• Garbage collection stall — the JuiceFS GC process (background data compaction) stops running; object storage usage grows unboundedly and performance degrades as fragmentation accumulates

External monitoring catches all three layers — metadata, object storage reachability, and CSI driver health — before applications notice.

What you'll need

• A running JuiceFS deployment (standalone or Kubernetes CSI)
• A free Vigilmon account (sign up takes 30 seconds)
• Network access to your metadata engine and optionally your object storage API

Step 1: Expose a health check endpoint for JuiceFS

JuiceFS exposes a built-in metrics and health endpoint when started with the --metrics flag. Enable it:

# Standalone mount with metrics
juicefs mount redis://your-redis:6379/1 /mnt/jfs \
  --metrics 0.0.0.0:9567 \
  --background

# Verify
curl http://localhost:9567/metrics | head -20

In Kubernetes, update your JuiceFS CSI values to expose the metrics port:

# juicefs-csi-values.yaml (Helm)
node:
  metricsPort: 9567
  resources:
    limits:
      cpu: 1
      memory: 1Gi

Create a NodePort service to expose the CSI node metrics externally:

# juicefs-metrics-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: juicefs-metrics
  namespace: kube-system
spec:
  type: NodePort
  selector:
    app: juicefs-csi-node
  ports:
    - name: metrics
      port: 9567
      targetPort: 9567
      nodePort: 30956
      protocol: TCP

kubectl apply -f juicefs-metrics-svc.yaml

# Verify metrics are accessible
curl http://<node-ip>:30956/metrics | grep juicefs_uptime

Step 2: Set up HTTP monitoring in Vigilmon

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS as the type
3. Set the URL to your JuiceFS metrics endpoint: http://your-node.example.com:30956/metrics
4. Set the check interval to 1 minute
5. Under Expected response, set:
   • Status code: 200
   • Response body contains: juicefs_uptime
6. Save the monitor

If the JuiceFS CSI node process crashes or the metrics server stops responding, Vigilmon triggers an alert immediately. The juicefs_uptime body check confirms you're seeing real JuiceFS metrics, not just a generic 200 from another service.

Step 3: Monitor the metadata engine

The metadata engine is the most critical component — a Redis or PostgreSQL failure makes JuiceFS completely non-functional. Monitor it directly:

Redis metadata engine

1. In Vigilmon, go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your Redis host and port: your-redis.example.com / 6379
4. Save the monitor

If you're using Redis Sentinel or Cluster, add TCP monitors for the Sentinel port (26379) and multiple cluster node ports as well.

PostgreSQL metadata engine

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your PostgreSQL host and port: your-postgres.example.com / 5432
4. Save the monitor

MinIO or custom object storage

If you're using MinIO rather than a managed object store, monitor its API endpoint:

1. Go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to http://your-minio.example.com:9000/minio/health/live
4. Expect status code 200
5. Save the monitor

Step 4: Monitor the JuiceFS CSI controller

The CSI controller handles volume provisioning. If it crashes, new PersistentVolumeClaim bindings for JuiceFS volumes will stall:

# Check CSI controller health
kubectl get pods -n kube-system -l app=juicefs-csi-controller

# Expose CSI controller metrics
kubectl port-forward -n kube-system \
  deployment/juicefs-csi-controller 9568:9568
curl http://localhost:9568/metrics | head -5

Create a NodePort for the CSI controller:

# juicefs-controller-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: juicefs-csi-controller-metrics
  namespace: kube-system
spec:
  type: NodePort
  selector:
    app: juicefs-csi-controller
  ports:
    - name: metrics
      port: 9568
      targetPort: 9568
      nodePort: 30957
      protocol: TCP

Add an HTTP monitor in Vigilmon for http://your-node.example.com:30957/metrics.

Step 5: Create a filesystem I/O heartbeat

The ultimate test of JuiceFS health is whether the filesystem can actually complete a read/write. Create a CronJob that performs a timed write and read:

# juicefs-io-heartbeat.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: juicefs-io-heartbeat
  namespace: storage-monitoring
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          volumes:
            - name: juicefs-vol
              persistentVolumeClaim:
                claimName: juicefs-heartbeat-pvc
          containers:
            - name: io-check
              image: alpine:latest
              volumeMounts:
                - name: juicefs-vol
                  mountPath: /mnt/jfs
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  
                  # Write probe file
                  echo "heartbeat-$(date -u +%s)" > /mnt/jfs/.heartbeat
                  
                  # Sync to storage
                  sync
                  
                  # Read it back and verify
                  CONTENT=$(cat /mnt/jfs/.heartbeat)
                  if echo "$CONTENT" | grep -q "heartbeat-"; then
                    curl -fsS "https://hb.vigilmon.online/your-heartbeat-token"
                    echo "JuiceFS I/O verified: $CONTENT"
                  else
                    echo "ERROR: read-back failed" >&2
                    exit 1
                  fi

Create the PVC for the heartbeat job:

# juicefs-heartbeat-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: juicefs-heartbeat-pvc
  namespace: storage-monitoring
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: juicefs-sc
  resources:
    requests:
      storage: 1Mi

In Vigilmon, create a Heartbeat monitor with a 10-minute interval. A JuiceFS hang, metadata engine outage, or object storage failure will cause the CronJob to timeout and stop pinging — triggering your alert.

Step 6: Configure alert channels

1. In Vigilmon, go to Alert Channels → Add Channel → Webhook
2. Enter your Slack, PagerDuty, or Opsgenie webhook URL
3. Assign all JuiceFS monitors to the channel

When a JuiceFS alert fires, run triage in this order:

# 1. Check CSI driver pods
kubectl get pods -n kube-system | grep juicefs

# 2. Check metadata engine reachability
redis-cli -h your-redis ping
# or: psql -h your-postgres -U juicefs -c "SELECT 1"

# 3. Look for hung mount processes
ps aux | grep "juicefs\|fuse"

# 4. Check for D-state processes (uninterruptible sleep on JuiceFS mounts)
ps aux | grep " D "

# 5. Check JuiceFS logs
kubectl logs -n kube-system -l app=juicefs-csi-node --tail=100

# 6. Check object storage reachability
curl -I http://your-minio:9000/minio/health/live
# or: aws s3 ls s3://your-juicefs-bucket/ --no-sign-request 2>&1 | head -5

Step 7: Create a public status page

If JuiceFS serves shared storage for multiple teams or production workloads:

1. In Vigilmon, go to Status Pages → New Status Page
2. Name it "Distributed Storage Infrastructure"
3. Group monitors by layer: Metadata, Object Storage, CSI Driver, I/O Heartbeat
4. Publish and share with teams that consume JuiceFS storage

Putting it all together

| Monitor | Type | What it catches | |---------|------|-----------------| | http://node:30956/metrics | HTTP | CSI node process crashes, metrics scrape failures | | http://node:30957/metrics | HTTP | CSI controller crashes, provisioning stalls | | your-redis:6379 | TCP | Metadata engine outage (instant JuiceFS failure) | | your-postgres:5432 | TCP | PostgreSQL metadata engine outage | | http://minio:9000/minio/health/live | HTTP | Object storage API unavailability | | I/O heartbeat CronJob | Heartbeat | End-to-end filesystem hangs, write/read failures |

What's next

• SSL certificate monitoring — if your metadata engine or object storage uses TLS, Vigilmon can alert you before certificates expire and cause connection failures
• Capacity planning alerts — export juicefs_object_pending_pages and juicefs_store_cache_used_bytes metrics to Prometheus and alert when cache hit rate drops below 80%
• Multi-region monitoring — if you replicate JuiceFS data across regions, create separate monitors for each regional metadata engine and aggregate them on a single status page

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.