tutorial

Monitoring Apache Samza with Vigilmon: Job Manager Health, Kafka Broker Connectivity, YARN/K8s Resource Manager & SSL Monitoring

How to monitor Apache Samza distributed stream processing with Vigilmon — job manager health checks, Kafka broker TCP monitoring, YARN ResourceManager or Kubernetes API uptime, and SSL certificate monitoring.

Apache Samza is the distributed stream processing framework from LinkedIn that handles stateful stream processing at scale — consuming from Kafka topics, maintaining local state via RocksDB, and producing processed results back to Kafka. When Samza's Job Coordinator goes down, all streaming tasks stop processing and partitions remain unconsumed. When Kafka broker connectivity is lost, both consumers and producers fail. When the YARN ResourceManager or Kubernetes control plane is unreachable, Samza cannot launch or restart failed containers. Vigilmon gives you external visibility into Samza's infrastructure dependencies: the Job Coordinator, Kafka broker connectivity, the cluster resource manager, and the REST APIs that expose job status.

What You'll Build

  • A TCP monitor on the Kafka broker port to detect the most critical Samza dependency failure
  • An HTTP monitor on the YARN ResourceManager or the Samza REST API to catch job coordinator failures
  • An SSL certificate monitor for your Kafka and REST API endpoints
  • Alerting that maps infrastructure failures to Samza-specific impact

Prerequisites

  • A running Samza application deployed on YARN, Kubernetes, or standalone mode
  • Kafka brokers accessible on the network with reachable hostnames/IPs and ports
  • YARN ResourceManager or Kubernetes API server accessible via HTTP/HTTPS
  • A free account at vigilmon.online

Step 1: Understand Samza's Infrastructure Dependencies

Samza applications depend on several external infrastructure components:

| Component | Default port | Role | |---|---|---| | Kafka broker | 9092 (plaintext), 9093 (TLS) | Input and output topic storage; checkpoint storage for offset management | | YARN ResourceManager | 8088 (HTTP), 8090 (HTTPS) | Container scheduling for Samza tasks (YARN deployments) | | YARN NodeManager | 8042 | Executes Samza containers on individual nodes | | Samza REST API | 9090 (if configured) | Job status, metrics, and management API | | Kubernetes API | 6443 | Container scheduling (Kubernetes deployments) | | ZooKeeper | 2181 | Leader election and coordination (older Samza versions) |

External monitoring prioritizes the Kafka broker (Samza's sole I/O path) and the cluster resource manager (which Samza relies on to recover from task failures by restarting containers).

Step 2: Monitor Kafka Broker Connectivity via TCP Check

Kafka is Samza's exclusive source of input streams and its primary output destination. Samza checkpoints consumer offsets to Kafka, uses Kafka for coordination in some configurations, and reads/writes all stream data through Kafka. A TCP check on the broker port is the most critical external monitor for a Samza deployment:

# Test Kafka broker TCP connectivity
nc -zv kafka.example.com 9092
# or for TLS-enabled brokers:
nc -zv kafka.example.com 9093
  1. Log in to Vigilmon → Add Monitor → TCP.
  2. Host: kafka.example.com.
  3. Port: 9092 (plaintext) or 9093 (TLS).
  4. Check interval: 60 seconds.
  5. Label: Kafka broker TCP (Samza).
  6. Click Save.

This monitor catches:

  • Kafka broker process crashes or cluster-wide failures
  • Network partitions between the Samza cluster and Kafka
  • Firewall rule changes blocking Kafka port access
  • Kafka cluster maintenance events that take brokers offline longer than expected

Alert sensitivity: Set to trigger after 1 consecutive failure. When Kafka becomes unreachable, Samza tasks cannot fetch input messages, cannot write output, and cannot checkpoint offsets — resulting in stalled processing and potential message reprocessing when connectivity is restored.

Monitor all seed brokers: Add a TCP monitor for each Kafka bootstrap broker in your Samza systems.<system>.samza.factory configuration. Samza can route around single broker failures, but monitoring individual brokers detects partial cluster degradation before it impacts processing throughput.

Step 3: Monitor YARN ResourceManager (YARN Deployments)

For Samza applications deployed on YARN, the ResourceManager is responsible for launching Samza's ApplicationMaster (Job Coordinator) and restarting failed task containers. If the ResourceManager is unavailable, failed Samza containers cannot be restarted:

# Check YARN ResourceManager cluster info API
curl http://yarn-rm.example.com:8088/ws/v1/cluster/info
# Returns JSON with cluster state, ResourceManager version, and availability
  1. Add Monitor → HTTP.
  2. URL: http://yarn-rm.example.com:8088/ws/v1/cluster/info (or HTTPS on port 8090).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: clusterInfo (present in the YARN cluster info JSON response).
  7. Label: YARN ResourceManager.
  8. Click Save.

If your YARN ResourceManager is configured with HA (High Availability with two ResourceManagers), add a monitor for each RM endpoint. A YARN RM HA failover causes a brief gap in container scheduling — monitoring both nodes helps identify which RM is the active leader and when a failover occurs.

Step 4: Monitor the Samza REST API (If Configured)

Samza includes an optional REST API (samza-rest) that exposes job status, task metrics, and management endpoints. If deployed, this API provides HTTP-accessible insight into job health:

# Check Samza REST API health
curl http://samza-api.example.com:9090/v1/jobs
# Returns JSON with running Samza jobs and their status
  1. Add Monitor → HTTP.
  2. URL: http://samza-api.example.com:9090/v1/jobs.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: jobs (present in the job list JSON response).
  7. Label: Samza REST API.
  8. Click Save.

The Samza REST API is an optional component and may not be deployed in all Samza installations. If not configured, skip this step and rely on the YARN ResourceManager and Kafka TCP monitors as your primary infrastructure health signals.

Step 5: Monitor Kubernetes API Server (Kubernetes Deployments)

For Samza deployments on Kubernetes (using the Samza Kubernetes integration or running Samza jobs as Kubernetes Jobs/Deployments), the Kubernetes API server is the critical control plane component. If the API server is unreachable, failed Samza pods cannot be rescheduled:

# Kubernetes API server health check
curl https://k8s-api.example.com:6443/healthz
# Returns "ok" when the API server is healthy
  1. Add Monitor → HTTP.
  2. URL: https://k8s-api.example.com:6443/healthz.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Kubernetes API server.
  8. Click Save.

Step 6: Monitor ZooKeeper Port (Legacy Coordination)

Older Samza deployments (before Kafka-based coordination was fully adopted) use ZooKeeper for leader election and offset management. If your Samza version uses ZooKeeper, a TCP check on the ZooKeeper port is essential:

# Test ZooKeeper connectivity
nc -zv zookeeper.example.com 2181
  1. Add Monitor → TCP.
  2. Host: zookeeper.example.com.
  3. Port: 2181.
  4. Check interval: 60 seconds.
  5. Label: ZooKeeper (Samza coordination).
  6. Click Save.

If your Samza version uses Kafka for coordination (via kafka.checkpoint.system), ZooKeeper is no longer a direct dependency for Samza and this step can be skipped. Check your job.coordinator.factory configuration to determine if ZooKeeper is still in use.

Step 7: Monitor SSL Certificates

Kafka brokers with TLS enabled and the Samza REST API (if HTTPS) require valid SSL certificates. An expired Kafka broker certificate causes all Samza consumers and producers to fail with TLS verification errors, stopping all stream processing:

# Check Kafka broker TLS certificate
openssl s_client -connect kafka.example.com:9093 2>/dev/null | openssl x509 -noout -dates

# Check Samza REST API certificate (if HTTPS)
openssl s_client -connect samza-api.example.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: kafka.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.
  6. Repeat for the Samza REST API domain if HTTPS.

Step 8: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Kafka broker TCP | Connection refused | Broker down; Samza processing stalled; check Kafka cluster health | | YARN ResourceManager | Non-200 or clusterInfo missing | RM unavailable; failed Samza containers won't restart; check YARN RM logs | | Samza REST API | Non-200 or jobs missing | REST API service down; check API server process | | Kubernetes API | Non-200 or ok missing | K8s control plane degraded; pod rescheduling will fail | | ZooKeeper TCP | Connection refused | ZooKeeper down; Samza coordination may fail (check if still in use) | | Kafka SSL certificate | < 30 days to expiry | Renew Kafka TLS cert; test Samza consumer/producer after renewal |

Alert after: 1 consecutive failure for the Kafka TCP and YARN RM monitors. 2 consecutive failures for the Samza REST API and ZooKeeper monitors.

Common Samza Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Kafka broker cluster failure | TCP monitor fires; all Samza processing stops; alert within 60 s | | YARN ResourceManager crash | RM HTTP monitor fires; failed containers not restarted; tasks accumulate lag | | Kafka broker TLS certificate expired | SSL monitor alerts; all TLS-enabled Samza tasks fail with verification errors | | Network partition between Samza cluster and Kafka | Kafka TCP monitor fires; processing stalls | | Kubernetes control plane degraded | K8s API monitor fires; pod rescheduling blocked | | ZooKeeper ensemble failure | ZooKeeper TCP monitor fires; Samza coordination breaks (if ZK-dependent) | | YARN RM HA failover (extended) | RM monitor catches downtime window between active/standby switch | | Kafka topic deleted (checkpoint topic) | Not directly caught by external monitoring; Samza tasks restart from earliest offset | | Samza ApplicationMaster OOM | Not caught externally; YARN restarts AM; visible in YARN ResourceManager UI | | DNS misconfiguration | All monitors fire simultaneously |

Monitoring Processing Health (Beyond Infrastructure)

Vigilmon monitors Samza's infrastructure dependencies — Kafka connectivity, the resource manager, and SSL certificates. Stream processing health requires application-level observability:

  • Kafka consumer lag: Monitor consumer group lag on Samza's input topics using Kafka's kafka-consumer-groups.sh tool, the Kafdrop UI, or Prometheus via the Kafka Exporter. Lag growth indicates Samza processing is falling behind.
  • Samza metrics: Samza publishes JMX metrics for task throughput, processing latency, and checkpoint lag. Integrate with Prometheus via JMX Exporter or configure Samza's MetricsReporter to send metrics to your monitoring stack.
  • Checkpoint monitoring: Monitor the age of Samza's most recent checkpoint in Kafka. Stale checkpoints mean that a restart will reprocess a large message window, increasing recovery time.
  • Container restarts: Monitor YARN ApplicationMaster restart count or Kubernetes pod restart count. Frequent restarts indicate application-level issues (OOM, uncaught exceptions) that aren't caught by infrastructure monitors.

Vigilmon catches infrastructure-level failures. Processing throughput, consumer lag, and checkpoint health require JMX metrics and Kafka-level lag monitoring.

Apache Samza is the distributed stream processor that handles high-throughput, stateful event processing at scale — when Kafka connectivity is lost or the resource manager goes down, Samza tasks stall silently, consumer offsets freeze, and downstream systems waiting for processed output stop receiving data. Vigilmon gives you external visibility into Samza's infrastructure before stalled processing becomes a data pipeline incident: Kafka broker TCP reachability, YARN or Kubernetes control plane health, ZooKeeper connectivity, and SSL certificate validity, so you know the moment the infrastructure supporting your stream processing jobs is threatened and can act before processing lag cascades into data quality issues.

Start monitoring Apache Samza in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles