tutorial

Spark Operator Monitoring on Kubernetes: Catch Failed Spark Jobs Before Your Data Pipeline Breaks

How to monitor Apache Spark on Kubernetes using the Spark Operator — detect job failures, driver pod crashes, and executor starvation with Vigilmon heartbeat and HTTP monitoring.

The Kubernetes Spark Operator lets you submit Spark jobs as SparkApplication CRDs rather than calling spark-submit manually. It handles driver pod lifecycle, executor scaling, and retry logic. But when a Spark job fails mid-execution, runs too slowly, or is never scheduled because the cluster is resource-starved, nothing proactively alerts your data engineering team.

Vigilmon adds the missing observability layer: heartbeat monitors that detect when Spark jobs stop completing on schedule, and HTTP monitors that verify the Spark History Server is available for post-mortem analysis when things go wrong.

Why Spark Operator Needs External Monitoring

The Spark Operator reconciles SparkApplication objects and updates their .status.applicationState.state field. But watching CRD status requires cluster access — your on-call engineer at 2 AM can't verify job status without kubectl. More importantly, these failure modes are invisible without external monitoring:

Job never starts. If the cluster has insufficient CPU or memory to schedule the driver pod, the SparkApplication sits in PENDING state indefinitely. No alert fires unless you explicitly watch for it.

Executor starvation. The driver starts successfully but can't acquire enough executors to make progress. The job runs — extremely slowly — until it eventually times out, hours later than expected.

Job completes with wrong output. A Spark job can reach COMPLETED state after writing partial or corrupt data if your business logic swallows exceptions. Heartbeat-based monitoring combined with data quality checks catches this.

Spark History Server is down. When a job fails and your team needs to inspect the Spark UI to diagnose the issue, a crashed History Server means lost event logs and a much longer incident response time.

Key Signals to Monitor

| Signal | Endpoint / Method | Failure Indicator | |--------|-------------------|-------------------| | Job completion | Heartbeat ping from job driver | Missing ping past grace window | | Spark History Server | http://<ingress>/ | Non-200 or timeout | | History Server API | http://<ingress>/api/v1/applications | Non-200 means API is broken | | Job submission API (if used) | Operator REST endpoint | Non-200 |

What You'll Set Up

  • Heartbeat monitor for each critical Spark pipeline
  • HTTP monitor for the Spark History Server
  • Alert routing to Slack
  • Recommended check intervals and grace periods per job cadence

You'll need a free Vigilmon account — no credit card required.

Step 1: Expose the Spark History Server

If you're running the Spark History Server on Kubernetes, expose it via Ingress:

# spark-history-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: spark-history-server
  namespace: spark-system
spec:
  ingressClassName: nginx
  rules:
    - host: spark-history.yourdomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: spark-history-server
                port:
                  number: 18080

Apply and verify:

kubectl apply -f spark-history-ingress.yaml
curl -s https://spark-history.yourdomain.com/api/v1/applications | head -c 200

Step 2: Monitor the Spark History Server

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://spark-history.yourdomain.com/api/v1/applications
  4. Check interval: 5 minutes (History Server is less critical than production endpoints)
  5. Expected status code: 200
  6. Response time threshold: 10000ms (History Server can be slow with large event logs)
  7. Save the monitor

Step 3: Heartbeat Monitoring for Spark Pipelines

The most important thing to monitor is whether your Spark jobs complete on schedule. Set up a heartbeat for each critical pipeline.

Create the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Name it: spark-etl-nightly (use your pipeline name)
  3. Expected interval: 24 hours (or match your pipeline schedule)
  4. Grace period: 2 hours (Spark jobs have high runtime variance)
  5. Save — copy the heartbeat URL

Add a Heartbeat Ping to Your SparkApplication Driver

The cleanest approach is adding a final step to your Spark job that pings Vigilmon after writing output. Here's a PySpark example:

# etl_job.py — run as a SparkApplication
import os
import urllib.request
from pyspark.sql import SparkSession

def main():
    spark = SparkSession.builder \
        .appName("NightlyETL") \
        .getOrCreate()

    # Your ETL logic
    df = spark.read.parquet("s3a://your-bucket/input/")
    result = df.filter(df.status == "active").groupBy("region").count()
    result.write.mode("overwrite").parquet("s3a://your-bucket/output/")

    spark.stop()

    # Ping Vigilmon only after successful write
    heartbeat_url = os.environ.get("VIGILMON_HEARTBEAT_URL")
    if heartbeat_url:
        urllib.request.urlopen(heartbeat_url, timeout=15)
        print("Vigilmon heartbeat sent.")

if __name__ == "__main__":
    main()

Wire the Heartbeat URL Into Your SparkApplication CRD

# sparkapplication.yaml
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: nightly-etl
  namespace: spark-system
spec:
  type: Python
  pythonVersion: "3"
  mode: cluster
  image: your-registry/spark-etl:latest
  mainApplicationFile: local:///app/etl_job.py
  sparkVersion: "3.5.0"
  driver:
    cores: 2
    memory: "4g"
    env:
      - name: VIGILMON_HEARTBEAT_URL
        valueFrom:
          secretKeyRef:
            name: vigilmon-secrets
            key: etl-heartbeat-url
  executor:
    cores: 4
    instances: 5
    memory: "8g"
  restartPolicy:
    type: OnFailure
    onFailureRetries: 2
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20

Store the heartbeat URL:

kubectl create secret generic vigilmon-secrets \
  --namespace spark-system \
  --from-literal=etl-heartbeat-url='https://vigilmon.online/heartbeat/your-unique-token'

Now if the ETL job fails after retries, gets stuck on executor acquisition, or is silently skipped by a broken scheduler, Vigilmon fires an alert when the expected heartbeat doesn't arrive.

Step 4: Handling Multiple Pipelines

For teams running many Spark pipelines, create one heartbeat monitor per critical job:

| Monitor Name | Interval | Grace | Pipeline | |---|---|---|---| | spark-nightly-etl | 24h | 2h | Nightly aggregation | | spark-hourly-features | 1h | 20min | Feature store refresh | | spark-weekly-model-data | 7d | 6h | Weekly ML training data |

Use Vigilmon's status page to group all Spark monitors in one dashboard and share it with your data engineering team.

Step 5: Alert Configuration

Slack Integration

  1. Create a Slack Incoming Webhook for your #data-pipeline-alerts channel
  2. In Vigilmon, go to Alert Channels → New Channel → Webhook
  3. Paste the webhook URL
  4. Assign it to all Spark monitors

On-Call for Critical Pipelines

For pipelines that feed production ML models or financial reporting:

  1. Add a PagerDuty alert channel for the heartbeat monitor of that specific pipeline
  2. Keep lower-priority pipelines (analytics, dashboards) on Slack only

Debugging a Missed Heartbeat

When Vigilmon fires a heartbeat alert, use this runbook to triage:

# 1. Check SparkApplication status
kubectl get sparkapplications -n spark-system

# 2. Inspect the failed application
kubectl describe sparkapplication nightly-etl -n spark-system | grep -A5 "Application State"

# 3. Check driver pod logs
kubectl logs -n spark-system -l spark-role=driver --tail=100

# 4. Check if executors were acquired
kubectl get pods -n spark-system -l spark-role=executor

# 5. Check node resource pressure
kubectl describe nodes | grep -A5 "Allocated resources"

If the driver pod never started, the issue is likely resource quotas or node capacity. If it started but executors are missing, check spot instance preemptions or resource limits.

Summary

The Spark Operator automates job submission and retry — but it cannot tell you when jobs are taking twice as long as usual, when the History Server is down during an incident, or when a job silently failed to write any output. Vigilmon provides that signal.

| Failure Mode | Spark Operator | Vigilmon | |---|---|---| | Driver pod crash | Retries per policy | Alerts via heartbeat if all retries fail | | Job never scheduled | Status: PENDING | Alerts when heartbeat window expires | | Executor starvation / slow job | ✗ | Alerts when heartbeat is late | | Silent data write failure | ✗ | Alerts via heartbeat (conditional ping) | | History Server down | ✗ | ✓ via HTTP monitor | | Ingress routing broken | ✗ | ✓ via HTTP monitor |

Set up your first Spark pipeline monitor free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles