tutorial

Argo Workflows Monitoring: Detect Failed and Stalled Pipelines Before They Break Production

How to monitor Argo Workflows on Kubernetes — UI availability, workflow completion heartbeats, CronWorkflow schedules, and artifact storage health — using Vigilmon external monitoring.

Argo Workflows is the leading workflow engine for Kubernetes — teams use it to orchestrate ML training pipelines, data ETL jobs, CI/CD flows, and multi-step batch processing. The Argo UI gives you a clear view of running and historical workflows when you're logged in. But when a CronWorkflow silently stops firing, when a critical pipeline fails at step 7 of 12 and your on-call engineer is asleep, or when the Argo Server itself goes down, you need an alert that reaches you without requiring a kubectl session.

Vigilmon gives Argo Workflows the external monitoring layer it lacks: HTTP checks that verify the Argo Server is reachable, and heartbeat monitors that tell you when workflows stop completing on schedule — without any Kubernetes access required to receive the alert.

Why Argo Workflows Needs External Monitoring

The Argo Workflows operator runs workflows as Pods with a sidecar wait container. It tracks status in CRDs. But several failure modes are invisible to Kubernetes-internal health checks:

Argo Server is down during an incident. When a critical pipeline fails and your team needs to inspect logs and parameters in the Argo UI, a crashed Argo Server or broken Ingress means no access to failure context — dramatically increasing mean time to resolution.

CronWorkflows silently stop firing. A CronWorkflow depends on the Argo Server's cron controller. After a controller upgrade, a misconfigured timezone, or a controller crash, cron schedules can stop firing indefinitely. The CronWorkflow object exists; no workflows run.

Workflow stalls in Running state. A workflow can enter a state where one step is waiting for a resource that will never come — a queue message, an external API, a full-disk artifact repository — and stays in Running status until a human notices it hasn't finished.

Artifact storage failure. Argo Workflows writes logs and artifacts to S3, GCS, or MinIO. If artifact storage fails, workflows hang at the upload step rather than completing. The failure is in your storage layer, not in Kubernetes — standard probes don't catch it.

Key Signals to Monitor

| Signal | Endpoint / Method | Failure Indicator | |--------|-------------------|-------------------| | Argo Server UI | https://argo.yourdomain.com/ | Non-200 or timeout | | Argo Server API | https://argo.yourdomain.com/api/v1/info | Non-200 | | CronWorkflow fires | Heartbeat ping from workflow step | Missing ping in schedule window | | Critical pipeline completes | Heartbeat ping from final step | Missing ping past SLA | | MinIO / artifact storage | http://minio.yourdomain.com/minio/health/live | Non-200 |

What You'll Set Up

  • HTTP monitor for the Argo Server
  • Heartbeat monitor for CronWorkflow completion
  • HTTP monitor for artifact storage health
  • Slack alert routing

You'll need a free Vigilmon account — no credit card required.

Step 1: Expose the Argo Server via Ingress

Argo Server should already be exposed externally for UI access. If not, create an Ingress:

# argo-server-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argo-server
  namespace: argo
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - argo.yourdomain.com
      secretName: argo-tls
  rules:
    - host: argo.yourdomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argo-server
                port:
                  number: 2746

Verify:

curl -sI https://argo.yourdomain.com/api/v1/info
# Expected: HTTP/2 200

Step 2: Monitor the Argo Server HTTP Endpoint

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://argo.yourdomain.com/api/v1/info
  4. Check interval: 1 minute
  5. Expected status code: 200
  6. Response body contains: "links" (confirms the API is serving)
  7. Response time threshold: 5000ms
  8. Save the monitor

This ensures your team can always access the Argo UI during incidents when debugging is most critical. It also catches Ingress routing failures and TLS certificate expiry that internal probes miss entirely.

Step 3: Heartbeat Monitoring for CronWorkflows

CronWorkflows are invisible to HTTP monitors — there's no endpoint to probe for "did this workflow run?" The solution is a heartbeat step in the workflow itself.

Create a Heartbeat Monitor in Vigilmon

  1. Go to Monitors → New Monitor → Heartbeat
  2. Name it: argo-ml-training-cron (use your workflow name)
  3. Expected interval: match the CronWorkflow schedule (e.g., 6 hours)
  4. Grace period: 1 hour (workflow runtime variance)
  5. Save — copy the heartbeat URL

Add a Vigilmon Ping Step to Your Workflow

Add a final template step that pings Vigilmon after all business logic completes:

# ml-training-workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: ml-training-pipeline
  namespace: argo
spec:
  schedule: "0 */6 * * *"
  timezone: "UTC"
  concurrencyPolicy: Forbid
  workflowSpec:
    entrypoint: pipeline
    templates:
      - name: pipeline
        steps:
          - - name: fetch-data
              template: fetch-data
          - - name: train-model
              template: train-model
          - - name: evaluate-model
              template: evaluate-model
          - - name: notify-vigilmon
              template: notify-vigilmon

      - name: fetch-data
        container:
          image: your-registry/data-fetcher:latest
          command: [python, /app/fetch.py]

      - name: train-model
        container:
          image: your-registry/trainer:latest
          command: [python, /app/train.py]
          resources:
            requests:
              nvidia.com/gpu: "1"
            limits:
              nvidia.com/gpu: "1"

      - name: evaluate-model
        container:
          image: your-registry/evaluator:latest
          command: [python, /app/evaluate.py]

      - name: notify-vigilmon
        container:
          image: curlimages/curl:latest
          command:
            - sh
            - -c
            - |
              curl -fsS "$VIGILMON_HEARTBEAT_URL"
              echo "Vigilmon heartbeat sent."
          env:
            - name: VIGILMON_HEARTBEAT_URL
              valueFrom:
                secretKeyRef:
                  name: vigilmon-secrets
                  key: ml-training-heartbeat-url
    volumes: []

Store the heartbeat URL as a Kubernetes Secret:

kubectl create secret generic vigilmon-secrets \
  --namespace argo \
  --from-literal=ml-training-heartbeat-url='https://vigilmon.online/heartbeat/your-unique-token'

Because the notify-vigilmon step is the last in the chain, it only runs when all previous steps succeed. If any earlier step fails, Argo marks the workflow as Failed and skips the ping — and Vigilmon alerts you when the heartbeat window expires.

Step 4: Monitor Critical One-Off Pipelines

For critical workflows that don't run on a cron schedule but have SLA requirements (e.g., a data pipeline that must complete within 4 hours of being triggered):

  1. Create a heartbeat monitor named argo-daily-report-pipeline
  2. Expected interval: 24 hours
  3. Grace period: 4 hours (matching your SLA)
  4. Add the same notify-vigilmon step to the workflow

This way even manually triggered workflows are covered — if someone triggers the pipeline and it fails or hangs, the heartbeat will alert within 4 hours.

Step 5: Monitor Artifact Storage

If Argo Workflows uses MinIO for artifact storage, add an HTTP monitor for MinIO's health endpoint. Expose it via Ingress:

# minio-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minio-health
  namespace: argo
spec:
  rules:
    - host: minio.yourdomain.com
      http:
        paths:
          - path: /minio/health/live
            pathType: Exact
            backend:
              service:
                name: minio
                port:
                  number: 9000

Add a Vigilmon monitor:

  • URL: https://minio.yourdomain.com/minio/health/live
  • Expected status: 200
  • Check interval: 2 minutes

When MinIO goes down, workflow artifact uploads will start failing — this monitor gives you 2-minute advance warning before workflows start timing out on the upload step.

Step 6: Status Page and Alert Routing

Create a Status Page

  1. Go to Status Pages → New Status Page
  2. Name it: "Workflow Infrastructure"
  3. Add all Argo monitors: Argo Server, CronWorkflow heartbeats, MinIO
  4. Share with your data engineering team

When users report issues, they can check the status page themselves before filing a ticket — saving your platform team triage time.

Configure Slack Alerts

  1. Create a Slack Incoming Webhook for #workflow-alerts
  2. In Vigilmon, go to Alert Channels → New Channel → Webhook
  3. Paste the webhook URL
  4. For the Argo Server HTTP monitor and MinIO monitor: assign immediately
  5. For heartbeat monitors: route to #pipeline-sla if you want SLA breaches tracked separately

Debugging a Missed Heartbeat

When Vigilmon fires a heartbeat alert for a workflow, use this triage sequence:

# 1. Find the most recent workflow run
kubectl get workflows -n argo --sort-by='.status.startedAt' | tail -10

# 2. Check workflow status and failed steps
kubectl describe workflow ml-training-pipeline-abc123 -n argo | grep -A5 "Phase"

# 3. Get logs from the failed step pod
kubectl logs -n argo -l workflows.argoproj.io/workflow=ml-training-pipeline-abc123

# 4. Check if the CronWorkflow is scheduling correctly
kubectl get cronworkflow ml-training-pipeline -n argo \
  -o jsonpath='{.status.lastScheduledTime}'

# 5. Check Argo controller logs for scheduling issues
kubectl logs -n argo -l app=workflow-controller --tail=50 | grep -i error

If lastScheduledTime is several cycles behind, the cron controller is not firing — this is the silent failure mode Vigilmon's heartbeat catches.

Recommended Thresholds

| Monitor | Check Interval | Alert After | Notes | |---------|---------------|-------------|-------| | Argo Server API | 1 min | 2 failures | Allow one transient failure | | CronWorkflow heartbeat | Per cron schedule | Grace: 1h | Adjust per workflow runtime | | MinIO health | 2 min | 1 failure | Storage failure = workflow hang | | One-off pipeline heartbeat | Per SLA | Grace: per SLA | Tune to business requirements |

Summary

Argo Workflows' internal status tracking is excellent for inspecting completed and running workflows — when you can access the Argo UI. Vigilmon ensures you can always access it, and that you know about failures before you open the UI.

| Failure Mode | Argo Operator | Vigilmon | |---|---|---| | Workflow step fails | Marks workflow Failed | Alerts via heartbeat (no ping sent) | | CronWorkflow stops firing | ✗ | ✓ via heartbeat (ping never arrives) | | Workflow hangs in Running | ✗ | ✓ via heartbeat (ping is late) | | Argo Server down | ✗ | ✓ via HTTP monitor | | MinIO artifact storage down | ✗ | ✓ via HTTP monitor | | Ingress or TLS broken | ✗ | ✓ via HTTP monitor |

Start monitoring your Argo Workflows pipelines free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles