tutorial

Monitoring Matillion with Vigilmon: Orchestration Agent Health, Repository API & Job Execution Availability

How to monitor Matillion cloud-native ETL/ELT pipelines with Vigilmon — Matillion agent health, repository API availability, cloud warehouse connectivity, and SSL certificate validity.

Matillion is a cloud-native ETL/ELT and data transformation platform designed to move and transform data at cloud scale using the compute power of modern cloud data warehouses — Snowflake, Redshift, BigQuery, Databricks, and others. Matillion's architecture separates an orchestration layer (the Matillion agent or server) from the transformation execution layer (pushed down to the target warehouse), which means data transformation jobs require both the Matillion orchestration infrastructure and the cloud warehouse to be healthy simultaneously. When the Matillion agent becomes unavailable, all scheduled jobs stop executing and no new transformations run — even if the target warehouse is perfectly healthy. When the Matillion repository API that stores job definitions and version history becomes inaccessible, developers cannot deploy new pipeline versions or retrieve the current state of their transformation logic. When the target warehouse loses connectivity from the Matillion agent, every job that attempts to push SQL to the warehouse fails with connection errors, and the Matillion scheduler queues jobs indefinitely or drops them depending on retry configuration. Vigilmon gives your data engineering team external HTTP visibility into each layer of the Matillion stack so that infrastructure failures are surfaced before they block transformation delivery.

What You'll Build

  • An HTTP monitor on the Matillion agent or server health endpoint to detect orchestration layer failures
  • A monitor on the Matillion repository API to catch degradation in job definition storage and deployment
  • A monitor on your target cloud warehouse endpoint to verify Matillion has compute resources to push transformations to
  • A monitor on your Matillion job failure webhook receiver to confirm alert delivery is operational
  • An SSL certificate monitor for the Matillion agent domain and warehouse API domain
  • Alerting that distinguishes orchestration failures from warehouse connectivity issues and notification breakdowns

Prerequisites

  • A Matillion instance deployed on AWS, Azure, or GCP (EC2, VM, or Kubernetes-based) or a Matillion SaaS workspace
  • A target cloud data warehouse configured as a Matillion environment (Snowflake, Redshift, BigQuery, or Databricks)
  • A webhook or notification endpoint configured to receive Matillion job failure notifications
  • A free account at vigilmon.online

Step 1: Understand Matillion's Infrastructure Architecture

Matillion's architecture has two primary layers: the Matillion orchestration agent (or server), which schedules and manages job execution, and the target cloud warehouse, which executes the pushed-down SQL transformations. Both layers must be healthy for transformation pipelines to run:

| Component | Surface | Role | |---|---|---| | Matillion agent / server | HTTPS REST API on port 443 or 8080 | Orchestrates job scheduling, manages environment connections, stores job state | | Matillion repository | API on the agent / SaaS backend | Stores job definitions, pipeline versions, and project metadata | | Target warehouse | Snowflake / Redshift / BigQuery API | Receives and executes pushed-down SQL transformations | | Job failure webhook | Your HTTPS endpoint | Receives Matillion job failure and task error notifications | | Matillion web UI | HTTPS | Engineer access during incidents; job logs, task graphs |

External monitoring focuses on the Matillion agent health endpoint, the repository API, and the target warehouse — these three surfaces must all be simultaneously available for transformation pipelines to execute successfully. A Matillion agent failure stops all job orchestration; a repository API failure blocks job deployments and version rollbacks; a warehouse connectivity failure causes all transformation tasks to fail at execution time.

Step 2: Monitor the Matillion Agent Health Endpoint

The Matillion agent is the orchestration core of every transformation pipeline — it schedules jobs, manages task execution order, handles retries, and reports job status back to the Matillion UI and API clients. When the agent becomes unavailable, all scheduled jobs stop executing and the entire transformation pipeline halts:

# Matillion agent health check (self-managed deployment)
curl https://your-matillion.example.com/rest/v1/server/info
# Returns: {"serverVersion": "...", "status": "ok"} when the agent is healthy

# For Matillion SaaS
curl https://your-account.matillion.com/api/v1/workspace/info
# Returns workspace details confirming API reachability
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://your-matillion.example.com/rest/v1/server/info.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: serverVersion (present in every healthy Matillion agent info response).
  7. Label: Matillion agent health.
  8. Click Save.

This monitor catches:

  • Matillion agent process crashes or OOM kills on the underlying VM or container
  • Infrastructure failures (EC2 instance stop, Azure VM deallocated, GCP VM preempted) that take the agent offline
  • Network changes that have isolated the Matillion agent from your monitoring infrastructure
  • Matillion agent upgrades that have left the service in a failed state post-restart

Alert sensitivity: Set to trigger after 1 consecutive failure. A dead Matillion agent means no transformation job will run, no matter how healthy the target warehouse is — the orchestration layer is the single dependency for all pipeline execution.

Step 3: Monitor the Matillion Repository API

The Matillion repository stores all job definitions, transformation pipeline logic, project configurations, and version history. When the repository API is unavailable, developers cannot deploy new pipeline versions, cannot roll back to a previous job definition in response to a production incident, and the Matillion agent may fail to load updated job configurations for pending scheduled runs:

# List projects via the Matillion API
curl -u admin:PASSWORD \
  https://your-matillion.example.com/rest/v1/project
# Returns: [{"id": 1, "name": "MyProject", ...}] when the repository API is accessible

# Fetch a specific project's job list
curl -u admin:PASSWORD \
  https://your-matillion.example.com/rest/v1/project/1/version/1/job
# Returns job list; confirms repository read path is healthy
  1. Add Monitor → HTTP.
  2. URL: https://your-matillion.example.com/rest/v1/project.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: name (present in every project list response when the repository is readable).
  7. Label: Matillion repository API.
  8. Click Save.

Why monitor the repository separately from the agent? In some Matillion deployments, the agent process can be alive and accepting health check requests while an internal database connection failure has degraded the repository layer — job submissions succeed at the API level but fail when Matillion attempts to read job definitions from the repository backend.

Step 4: Monitor the Target Cloud Warehouse Endpoint

Matillion pushes transformation SQL to your cloud warehouse for execution. The warehouse must be reachable from the Matillion agent for any transformation task to run. A warehouse that is unavailable, suspended, or returning errors causes every Matillion task to fail at execution time — but the Matillion agent itself stays healthy, which means the failure only becomes visible through job error logs rather than agent health:

# Snowflake account reachability
curl https://your-account.snowflakecomputing.com/session/v1/login-request
# Returns: {"data": null, "code": null} — confirms Snowflake endpoint is reachable (auth failure expected)

# Redshift cluster endpoint (via a health-check service or load balancer)
curl https://your-redshift-health.example.com/health
# Returns: {"status": "ok"} when Redshift is accepting connections

# BigQuery API availability
curl https://bigquery.googleapis.com/$discovery/rest?version=v2
# Returns: BigQuery API discovery document when the API is reachable
  1. Add Monitor → HTTP.
  2. URL: https://your-account.snowflakecomputing.com/session/v1/login-request (or the appropriate health endpoint for your warehouse type).
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: data (present in the Snowflake login-request response body; adjust for your warehouse).
  7. Label: Matillion target warehouse (Snowflake/Redshift/BigQuery).
  8. Click Save.

This monitor gives your team advance warning that the warehouse is degraded or suspended before Matillion begins queuing failed transformation tasks, allowing you to investigate warehouse availability before job failure backlog accumulates.

Step 5: Monitor Your Job Failure Webhook Receiver

Matillion can send job completion and failure notifications to external webhooks using the Matillion notification API or via task-embedded Python/bash scripts that post to your alerting infrastructure. When this receiver is down, your team receives no signal when a Matillion transformation job fails, and data consumers discover pipeline failures through stale dashboard data rather than proactive alerts:

curl https://your-matillion-alerts.example.com/health
# Returns: {"status": "ok"} when the job failure receiver is operational
  1. Add Monitor → HTTP.
  2. URL: https://your-matillion-alerts.example.com/health.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Matillion job failure webhook.
  8. Click Save.

For Slack-integrated Matillion failure notifications, verify https://slack.com/api/api.test to confirm Slack API reachability from your alerting infrastructure. For PagerDuty-routed Matillion incidents, monitor https://api.pagerduty.com/ability to verify the Events API is accessible. If Matillion job failures feed into a data observability platform like Monte Carlo or Bigeye, add a health endpoint monitor for that platform's ingestion API as well.

Step 6: Monitor SSL Certificates

The Matillion agent is accessed by data engineers via HTTPS for job management, and by automated CI/CD pipelines that deploy job definitions using the Matillion REST API. An expired certificate on the Matillion agent domain locks out both human engineers during incident response and automated deployment pipelines that push new job versions — the CI/CD pipeline fails to deploy the hotfix exactly when a failed pipeline most needs it:

openssl s_client -connect your-matillion.example.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: your-matillion.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Repeat for your cloud warehouse domain (your-account.snowflakecomputing.com, your-redshift-cluster.region.redshift.amazonaws.com, etc.) if you manage TLS for those endpoints, and for any internal services that the Matillion agent calls during transformation execution. Self-managed Matillion deployments on EC2 or Azure VMs often use Let's Encrypt certificates with 90-day expiry — Vigilmon's 30-day alert threshold provides a two-renewal-cycle window.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Matillion agent health | Non-200 or serverVersion missing | No transformation jobs will execute; escalate to infra team | | Matillion repository API | Non-200 or name missing | Job deployment and rollback blocked; pipeline versions locked | | Target warehouse endpoint | Non-200 or keyword missing | All Matillion transformation tasks will fail at execution | | Job failure webhook | Non-200 or ok missing | Matillion failure notifications not being delivered | | SSL certificate | < 30 days to expiry | Renew certificate; CI/CD deployment and engineer access blocked |

Alert after: 1 consecutive failure for the Matillion agent and repository API. 2 consecutive failures for the warehouse endpoint and webhook receiver to reduce noise from brief transient timeouts during warehouse auto-suspend/resume cycles.

Common Matillion Infrastructure Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Matillion EC2/VM instance stopped | Agent health monitor fires; all jobs halt | | OOM kill of Matillion agent process | Agent health monitor fires with connection refused | | Repository backend database failure | Repository API monitor fires; job deployments blocked | | Snowflake warehouse suspended | Warehouse monitor catches reachability; all tasks fail | | Redshift cluster unavailable | Warehouse monitor fires; transformation execution fails | | Webhook receiver crash | Alert monitor fires; job failure notifications lost | | TLS certificate expired on Matillion agent | SSL monitor alerts at 30-day threshold | | Network security group change blocks Matillion API | Agent and repository monitors fire with timeout | | CI/CD pipeline fails to deploy new job version | Repository API monitor surfaces if API is degraded | | Matillion agent upgrade leaves service degraded | Agent health monitor fires during upgrade window |

Monitoring Transformation Logic (Beyond Infrastructure)

Vigilmon monitors the infrastructure that Matillion runs on — the orchestration agent, the repository API, and the target warehouse connectivity. The correctness and performance of the transformation logic that Matillion executes requires domain-specific observability within the Matillion platform and your data warehouse:

  • Job run duration monitoring: Track the wall-clock time of individual Matillion job runs using the Matillion API's run history endpoint. Jobs that are consistently running slower than historical baselines indicate data volume growth or warehouse performance degradation — invisible to infrastructure monitors.
  • Task-level failure rate tracking: Use the Matillion API to query failed task counts per job over rolling time windows. A job that passes its infrastructure health checks but has a rising task failure rate indicates a source schema change, a warehouse permission change, or a transformation logic bug.
  • Row count validation in transformation outputs: After Matillion writes transformed data to your warehouse, validate row counts and key statistics against expected values from upstream source tables. Matillion pipeline infrastructure health does not tell you that a transformation join condition has silently changed output cardinality.

Vigilmon catches infrastructure failures — when the Matillion agent is down, the repository API is unreachable, or the target warehouse has lost connectivity. The accuracy of the SQL transformations, the completeness of the data written to your warehouse tables, and the performance characteristics of individual Matillion jobs require monitoring within the Matillion platform and your cloud data warehouse.

Matillion brings cloud-native ETL/ELT to modern data warehouses, but the infrastructure it runs on — the orchestration agent, the repository backend, and the target warehouse connectivity — can fail in ways that silently stop transformation pipelines without surfacing visible errors to data consumers. Vigilmon gives your data engineering team external visibility into that infrastructure: agent health, repository API availability, warehouse reachability, and SSL certificate validity, so you know the moment the Matillion platform needs attention and can restore transformation execution before data freshness SLAs are missed.

Start monitoring your Matillion infrastructure in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles