How to Monitor Humanitec with Vigilmon

Humanitec is a platform engineering platform that manages dynamic configuration, environment provisioning, and deployment automation for development teams. Platform engineers use Humanitec to define Resource Definitions — databases, message queues, DNS, and other infrastructure resources — that developers can request and consume without writing infrastructure code directly. Humanitec's deployment pipeline integrates with Kubernetes, Terraform, and dozens of cloud providers to automate the full lifecycle of application environments. When Humanitec is unavailable or its resource drivers are misbehaving, development teams cannot deploy to any environment, cannot provision new resources, and cannot trust that their running environments are in their declared state.

Vigilmon monitors Humanitec's API availability, deployment pipeline health, and resource driver liveness so platform engineering teams know immediately when the deployment platform itself has a problem.

Why Humanitec Needs External Monitoring

Humanitec runs as a SaaS control plane, but its integration layer — resource drivers, webhooks, and agent deployments — runs in your infrastructure and has several potential failure modes:

• Humanitec API unavailability — the control plane is SaaS but API downtime blocks all deployments and resource provisioning
• Resource driver failures — drivers for AWS, GCP, Kubernetes, and custom resources run as components in your cluster; when they fail, resource requests hang indefinitely
• Deployment pipeline stalls — a deployment stuck in pending or deploying state can block all subsequent deployments to an environment
• Operator health — the Humanitec Operator runs in your Kubernetes clusters and applies resource definitions; when the operator goes down, no configuration changes reach the cluster
• Webhook delivery failures — Humanitec sends webhooks for deployment state changes; failed delivery breaks CI/CD integrations downstream

Vigilmon monitors the Humanitec API and your integration infrastructure so deployment platform failures surface immediately rather than when a developer reports they cannot deploy.

Step 1: Verify Humanitec API Access

Humanitec exposes a REST API at api.humanitec.io. Verify your API token and basic connectivity:

# List organizations (basic API health check)
curl -s -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  'https://api.humanitec.io/orgs' | jq '.[] | .id'

# List applications in your organization
curl -s -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  "https://api.humanitec.io/orgs/$ORG_ID/apps" | jq '.[].id'

# Get active deployments across environments
curl -s -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  "https://api.humanitec.io/orgs/$ORG_ID/apps/$APP_ID/envs/production/deploys?count=1" | jq .

A 200 response confirms API access. If you get 401, regenerate the API token in the Humanitec dashboard.

Step 2: Build a Health Proxy for Vigilmon

Humanitec's API requires bearer token authentication. Expose a health proxy that checks the API and deployment state:

Node.js Health Proxy

// humanitec-health.js
const express = require('express');
const axios = require('axios');

const app = express();

const HUMANITEC_TOKEN = process.env.HUMANITEC_TOKEN;
const HUMANITEC_ORG = process.env.HUMANITEC_ORG;
const HUMANITEC_API = 'https://api.humanitec.io';

const humanitecApi = axios.create({
  baseURL: HUMANITEC_API,
  headers: { Authorization: `Bearer ${HUMANITEC_TOKEN}` },
  timeout: 10000,
});

app.get('/health/humanitec', async (req, res) => {
  try {
    // 1. Check org access (API availability)
    const orgsRes = await humanitecApi.get('/orgs');
    const orgIds = orgsRes.data?.map(o => o.id) ?? [];
    if (!orgIds.includes(HUMANITEC_ORG)) {
      return res.status(503).json({
        status: 'down',
        reason: 'org_not_accessible',
        availableOrgs: orgIds,
      });
    }

    // 2. List apps (confirms org data access)
    const appsRes = await humanitecApi.get(`/orgs/${HUMANITEC_ORG}/apps`);
    const appCount = appsRes.data?.length ?? 0;

    return res.status(200).json({
      status: 'ok',
      org: HUMANITEC_ORG,
      appCount,
    });
  } catch (err) {
    const status = err.response?.status;
    if (status === 401 || status === 403) {
      return res.status(503).json({ status: 'down', reason: 'authentication_failed', httpStatus: status });
    }
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3013, () => console.log('Humanitec health proxy on :3013'));

Python Health Proxy (FastAPI)

# humanitec_health.py
from fastapi import FastAPI
from fastapi.responses import JSONResponse
import httpx, os

app = FastAPI()

HUMANITEC_TOKEN = os.environ["HUMANITEC_TOKEN"]
HUMANITEC_ORG = os.environ["HUMANITEC_ORG"]
HUMANITEC_API = "https://api.humanitec.io"

@app.get("/health/humanitec")
async def humanitec_health():
    headers = {"Authorization": f"Bearer {HUMANITEC_TOKEN}"}

    async with httpx.AsyncClient(timeout=10.0) as client:
        try:
            orgs = await client.get(f"{HUMANITEC_API}/orgs", headers=headers)
            orgs.raise_for_status()
            org_ids = [o["id"] for o in orgs.json()]

            if HUMANITEC_ORG not in org_ids:
                return JSONResponse(status_code=503, content={
                    "status": "down", "reason": "org_not_accessible"
                })

            apps = await client.get(
                f"{HUMANITEC_API}/orgs/{HUMANITEC_ORG}/apps",
                headers=headers,
            )
            apps.raise_for_status()
            app_count = len(apps.json())

            return {"status": "ok", "org": HUMANITEC_ORG, "appCount": app_count}

        except httpx.HTTPStatusError as e:
            if e.response.status_code in (401, 403):
                return JSONResponse(status_code=503, content={
                    "status": "down", "reason": "authentication_failed"
                })
            return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})
        except Exception as e:
            return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Step 3: Monitor Deployment Pipeline Health

Stuck deployments are a common Humanitec failure mode. A deployment that never transitions from in_progress to succeeded or failed blocks all subsequent deployments to that environment. Monitor for stuck deployments:

// Add to humanitec-health.js
const MONITORED_APPS = (process.env.MONITORED_APPS || '').split(',').filter(Boolean);
const MONITORED_ENVS = (process.env.MONITORED_ENVS || 'production,staging').split(',');
const MAX_DEPLOY_AGE_MINUTES = parseInt(process.env.MAX_DEPLOY_AGE_MINUTES || '30');

app.get('/health/humanitec/deployments', async (req, res) => {
  const stuckDeployments = [];
  const checkedEnvs = [];

  for (const appId of MONITORED_APPS) {
    for (const envId of MONITORED_ENVS) {
      try {
        const deploysRes = await humanitecApi.get(
          `/orgs/${HUMANITEC_ORG}/apps/${appId}/envs/${envId}/deploys?count=1`
        );

        const deploys = deploysRes.data ?? [];
        if (deploys.length === 0) continue;

        const latest = deploys[0];
        checkedEnvs.push({ app: appId, env: envId, status: latest.status });

        if (latest.status === 'in_progress') {
          const startedAt = new Date(latest.created_at);
          const ageMinutes = (Date.now() - startedAt.getTime()) / (1000 * 60);

          if (ageMinutes > MAX_DEPLOY_AGE_MINUTES) {
            stuckDeployments.push({
              app: appId,
              env: envId,
              deployId: latest.id,
              status: latest.status,
              ageMinutes: Math.round(ageMinutes),
              startedAt: startedAt.toISOString(),
            });
          }
        }
      } catch (err) {
        // Skip envs that don't exist for this app
        if (err.response?.status !== 404) {
          checkedEnvs.push({ app: appId, env: envId, error: err.message });
        }
      }
    }
  }

  if (stuckDeployments.length > 0) {
    return res.status(503).json({
      status: 'degraded',
      reason: 'stuck_deployments_detected',
      stuckDeployments,
      checkedEnvs,
    });
  }

  return res.status(200).json({ status: 'ok', checkedEnvs });
});

Set MONITORED_APPS=my-app,backend-api and MAX_DEPLOY_AGE_MINUTES=30 to catch deployments that run past their expected completion window.

Step 4: Monitor Humanitec Operator Health in Kubernetes

The Humanitec Operator applies resource definitions from the control plane to your Kubernetes clusters. If the operator pod is unhealthy, no configuration changes are applied:

// humanitec-operator-health.js
// Run this as a sidecar in the same cluster as the Humanitec Operator
const express = require('express');
const { execSync } = require('child_process');

const app = express();

const OPERATOR_NAMESPACE = process.env.OPERATOR_NAMESPACE || 'humanitec-operator-system';
const OPERATOR_LABEL = process.env.OPERATOR_LABEL || 'app.kubernetes.io/name=humanitec-operator';

app.get('/health/humanitec/operator', (req, res) => {
  try {
    // Check operator pod status via kubectl
    const output = execSync(
      `kubectl get pods -n ${OPERATOR_NAMESPACE} -l ${OPERATOR_LABEL} ` +
      '--no-headers -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,READY:.status.containerStatuses[0].ready',
      { timeout: 10000, encoding: 'utf8' }
    );

    const lines = output.trim().split('\n').filter(Boolean);
    if (lines.length === 0) {
      return res.status(503).json({
        status: 'down',
        reason: 'no_operator_pods_found',
        namespace: OPERATOR_NAMESPACE,
      });
    }

    const pods = lines.map(line => {
      const [name, phase, ready] = line.trim().split(/\s+/);
      return { name, phase, ready: ready === 'true' };
    });

    const unhealthyPods = pods.filter(p => p.phase !== 'Running' || !p.ready);

    if (unhealthyPods.length === pods.length) {
      return res.status(503).json({
        status: 'down',
        reason: 'all_operator_pods_unhealthy',
        pods,
      });
    }

    if (unhealthyPods.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'some_operator_pods_unhealthy',
        pods,
        unhealthyPods,
      });
    }

    return res.status(200).json({ status: 'ok', pods });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3014, () => console.log('Humanitec operator health on :3014'));

Step 5: Configure Vigilmon HTTP Monitors

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your Humanitec health proxy: https://your-app.example.com/health/humanitec
4. Set the check interval to 2 minutes
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 10000ms
6. Under Alert channels, assign your platform engineering Slack and PagerDuty
7. Save the monitor

Add a deployment pipeline monitor:

• URL: https://your-app.example.com/health/humanitec/deployments
• Expected: 200, body contains "status":"ok"
• Interval: 5 minutes
• Alert channel: platform engineering Slack

Add an operator health monitor (if you expose it through Ingress):

• URL: https://monitoring.cluster.example.com/health/humanitec/operator
• Expected: 200, body contains "status":"ok"
• Interval: 2 minutes
• Alert channel: platform engineering on-call

Step 6: Heartbeat Monitoring for Deployment Pipeline Continuity

Confirm that successful deployments continue to happen on your most active environments. If no deployments complete within a rolling window, it may indicate that the pipeline is silently stuck or developers have stopped deploying due to repeated failures:

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: humanitec-staging-deployments-active
3. Set the expected interval: 2 hours (adjust to your team's deployment frequency)
4. Set the grace period: 4 hours
5. Save — copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123xyz

Deployment Completion Monitor

// humanitec-deploy-heartbeat.js
const axios = require('axios');

const HUMANITEC_TOKEN = process.env.HUMANITEC_TOKEN;
const HUMANITEC_ORG = process.env.HUMANITEC_ORG;
const HUMANITEC_APP = process.env.HUMANITEC_APP;
const HEARTBEAT_URL = process.env.VIGILMON_HEARTBEAT_URL;
const CHECK_ENV = process.env.CHECK_ENV || 'staging';
const CHECK_INTERVAL_MS = 30 * 60 * 1000; // 30 minutes

const api = axios.create({
  baseURL: 'https://api.humanitec.io',
  headers: { Authorization: `Bearer ${HUMANITEC_TOKEN}` },
  timeout: 10000,
});

let lastSeenDeployId = null;

async function checkDeploymentActivity() {
  const deploysRes = await api.get(
    `/orgs/${HUMANITEC_ORG}/apps/${HUMANITEC_APP}/envs/${CHECK_ENV}/deploys?count=1`
  );

  const deploys = deploysRes.data ?? [];
  if (deploys.length === 0) {
    console.log(`No deployments found for ${HUMANITEC_APP}/${CHECK_ENV}`);
    return;
  }

  const latest = deploys[0];

  if (latest.status === 'succeeded' && latest.id !== lastSeenDeployId) {
    lastSeenDeployId = latest.id;

    if (HEARTBEAT_URL) {
      await axios.get(HEARTBEAT_URL, { timeout: 3000 }).catch(() => {});
    }
    console.log(`Deployment activity confirmed: ${latest.id} succeeded`);
  } else {
    console.log(`Latest deploy: ${latest.id} (status: ${latest.status}) — no new success`);
  }
}

checkDeploymentActivity().catch(console.error);
setInterval(() => checkDeploymentActivity().catch(console.error), CHECK_INTERVAL_MS);

Step 7: Alert Routing for Humanitec Failures

| Monitor | Alert Channel | Priority | |---|---|---| | /health/humanitec (API availability) | Slack + PagerDuty | P1 | | /health/humanitec/deployments (stuck deployments) | Slack + email | P1 | | /health/humanitec/operator (operator pod health) | Slack + PagerDuty | P1 | | Heartbeat: staging-deployments-active | Slack + email | P2 | | Heartbeat: production-deployments-active | Slack + PagerDuty | P2 |

Stuck deployments in production warrant P1 classification because they can block emergency hotfixes and rollbacks. The deployment pipeline monitor should alert immediately (0-minute grace period) for production environments, with a 30-minute grace period for non-critical environments.

The Humanitec Operator is your highest-priority monitor after the API itself — if the operator is down, no infrastructure changes reach your clusters, and your environments silently drift from their declared state without any user-visible error.

Configure Vigilmon's response time threshold on the operator health check to 5000ms. A slow operator is often the first symptom of a resource-exhausted node before the operator pod is evicted.

Summary

Humanitec powers your internal developer platform's deployment automation, environment provisioning, and resource management. When Humanitec has problems, developers cannot deploy and platform engineers cannot fulfill resource requests. Vigilmon gives you the outside-in monitoring that keeps your deployment platform reliable:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/humanitec | Humanitec API availability and org access | | HTTP monitor on /health/humanitec/deployments | Stuck deployment detection across environments | | HTTP monitor on /health/humanitec/operator | Kubernetes Operator pod health | | Heartbeat: deployment-activity | End-to-end deployment pipeline continuity |

Get started free at vigilmon.online — your first Humanitec platform monitor is live in under two minutes.