health_felix.py

title: "How to Monitor Apache Felix OSGi Container Health with Vigilmon" published: false description: Apache Felix bundle lifecycle failures, fragment host mismatches, and service registry gaps are invisible to HTTP uptime checks. Learn how to monitor Felix bundle state, OSGi service availability, and container liveness with Vigilmon HTTP probes and heartbeat monitors. tags: felix, osgi, java, monitoring, bundles, devops cover_image:

Apache Felix is the reference implementation of the OSGi specification — a dynamic component model for Java that allows bundles (JARs with OSGi metadata) to be installed, started, stopped, and uninstalled at runtime without restarting the JVM. This dynamism is Felix's greatest strength and its primary monitoring challenge: a bundle can fail to resolve its dependencies at startup, enter the INSTALLED state instead of ACTIVE, and silently degrade services that depend on it — all while the JVM remains running and HTTP health checks return 200.

Vigilmon gives you external visibility into Felix container health through HTTP probe monitoring and heartbeat monitors. This tutorial covers both.

Why Felix Needs External Monitoring

Felix's OSGi lifecycle introduces failure modes that standard JVM monitoring misses:

• Bundle resolution failures: if a bundle declares an Import-Package that no other bundle exports at the required version, it stays in INSTALLED state indefinitely — services it provides are simply absent, with no error returned to callers
• Fragment host mismatches: OSGi fragment bundles attach to a host bundle; if the host version changes, the fragment stays unattached and silently drops any resources or classes it was contributing
• Declarative Services (SCR) component failures: OSGi DS components activate only when all @Reference dependencies are satisfied; a missing dependency keeps a component in UNSATISFIED state — callers that use ServiceTracker get null without an exception
• Bundle wiring conflicts: two bundles exporting the same package at different versions cause a split-package situation; Felix's resolver picks one wire, but the losing bundle's classes are silently shadowed
• Configuration Admin missing: many Felix applications depend on ConfigurationAdmin for runtime configuration; if the org.apache.felix.configadmin bundle fails to start, all DS components using @Activate(configurationPolicy = REQUIRE) remain inactive

External monitoring with Vigilmon adds:

• Proactive alerting when any bundle is not in the ACTIVE state at startup
• Service registry monitoring to detect missing OSGi service registrations
• Heartbeat monitoring to catch silent service failures invisible to bundle state checks
• Multi-region HTTP probing from outside the Felix container

Step 1: Build a Felix Health Endpoint

Felix includes the Web Console (org.apache.felix.webconsole) which exposes bundle state at /system/console/bundles.json. For production health checks, expose a lightweight custom endpoint instead.

Felix Servlet Component (Declarative Services)

// HealthServlet.java
import org.apache.felix.scr.annotations.*;
import org.osgi.framework.*;
import org.osgi.service.component.annotations.*;
import org.osgi.service.http.HttpService;
import javax.servlet.http.*;
import javax.servlet.*;
import java.io.*;
import java.util.*;

@Component(service = HealthServlet.class, immediate = true)
public class HealthServlet extends HttpServlet {

    @Reference
    private HttpService httpService;

    @Reference
    private BundleContext bundleContext;

    @Activate
    protected void activate() throws Exception {
        httpService.registerServlet("/health/felix", this, null, null);
    }

    @Deactivate
    protected void deactivate() {
        httpService.unregister("/health/felix");
    }

    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws IOException {
        resp.setContentType("application/json");

        Bundle[] bundles = bundleContext.getBundles();
        List<Map<String, Object>> problemBundles = new ArrayList<>();

        for (Bundle bundle : bundles) {
            int state = bundle.getState();
            // ACTIVE = 32, RESOLVED = 4 (fragments), FRAGMENT = ok if attached
            boolean isFragment = bundle.getHeaders().get("Fragment-Host") != null;
            boolean ok = state == Bundle.ACTIVE || (isFragment && state == Bundle.RESOLVED);

            if (!ok) {
                Map<String, Object> info = new LinkedHashMap<>();
                info.put("id", bundle.getBundleId());
                info.put("symbolic_name", bundle.getSymbolicName());
                info.put("version", bundle.getVersion().toString());
                info.put("state", stateToString(state));
                problemBundles.add(info);
            }
        }

        resp.setStatus(problemBundles.isEmpty() ? 200 : 503);
        StringBuilder json = new StringBuilder("{");
        json.append("\"status\":\"").append(problemBundles.isEmpty() ? "ok" : "degraded").append("\",");
        json.append("\"total_bundles\":").append(bundles.length).append(",");
        json.append("\"problem_count\":").append(problemBundles.size());
        json.append("}");
        resp.getWriter().write(json.toString());
    }

    private String stateToString(int state) {
        return switch (state) {
            case Bundle.UNINSTALLED -> "UNINSTALLED";
            case Bundle.INSTALLED -> "INSTALLED";
            case Bundle.RESOLVED -> "RESOLVED";
            case Bundle.STARTING -> "STARTING";
            case Bundle.STOPPING -> "STOPPING";
            case Bundle.ACTIVE -> "ACTIVE";
            default -> "UNKNOWN(" + state + ")";
        };
    }
}

Karaf-Hosted Felix Health Command (Felix in Apache Karaf)

// HealthCommand.java — Apache Karaf shell command
import org.apache.karaf.shell.api.action.*;
import org.apache.karaf.shell.api.action.lifecycle.Service;
import org.osgi.framework.*;

@Command(scope = "health", name = "felix-check",
         description = "Check Felix bundle health for Vigilmon")
@Service
public class HealthCommand implements Action {

    @Reference
    BundleContext bundleContext;

    @Override
    public Object execute() throws Exception {
        Bundle[] bundles = bundleContext.getBundles();
        long inactive = Arrays.stream(bundles)
            .filter(b -> {
                boolean isFragment = b.getHeaders().get("Fragment-Host") != null;
                return b.getState() != Bundle.ACTIVE &&
                    !(isFragment && b.getState() == Bundle.RESOLVED);
            })
            .count();

        System.out.printf("{\"status\":\"%s\",\"inactive_bundles\":%d,\"total_bundles\":%d}%n",
            inactive == 0 ? "ok" : "degraded", inactive, bundles.length);
        return null;
    }
}

Node.js Sidecar (Polling Felix Web Console)

// health/felix.js
const express = require('express');
const http = require('http');

const app = express();
const FELIX_HOST = process.env.FELIX_HOST || 'localhost';
const FELIX_PORT = parseInt(process.env.FELIX_PORT || '8080');
const FELIX_USER = process.env.FELIX_USER || 'admin';
const FELIX_PASS = process.env.FELIX_PASS || 'admin';

function fetchBundles() {
  return new Promise((resolve, reject) => {
    const auth = Buffer.from(`${FELIX_USER}:${FELIX_PASS}`).toString('base64');
    const req = http.get(
      {
        host: FELIX_HOST,
        port: FELIX_PORT,
        path: '/system/console/bundles.json',
        headers: { Authorization: `Basic ${auth}` },
        timeout: 8000,
      },
      res => {
        let body = '';
        res.on('data', c => { body += c; });
        res.on('end', () => resolve({ status: res.statusCode, body }));
      }
    );
    req.on('error', reject);
    req.on('timeout', () => { req.destroy(); reject(new Error('timeout')); });
  });
}

app.get('/health/felix', async (req, res) => {
  try {
    const result = await fetchBundles();
    if (result.status !== 200) {
      return res.status(503).json({ status: 'down', console_status: result.status });
    }

    const data = JSON.parse(result.body);
    const total = data.data ? data.data.length : 0;
    const inactive = (data.data || []).filter(b => {
      const isFragment = (b.fragment === true);
      return b.state !== 'Active' && !(isFragment && b.state === 'Resolved');
    }).length;

    const ok = inactive === 0;
    return res.status(ok ? 200 : 503).json({
      status: ok ? 'ok' : 'degraded',
      total_bundles: total,
      inactive_bundles: inactive,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3013);

Python Health Sidecar

import os, requests
from requests.auth import HTTPBasicAuth
from flask import Flask, jsonify

app = Flask(__name__)

FELIX_BASE = f"http://{os.environ.get('FELIX_HOST', 'localhost')}:{os.environ.get('FELIX_PORT', '8080')}"
FELIX_AUTH = HTTPBasicAuth(
    os.environ.get('FELIX_USER', 'admin'),
    os.environ.get('FELIX_PASS', 'admin')
)

@app.route('/health/felix')
def health():
    try:
        resp = requests.get(f'{FELIX_BASE}/system/console/bundles.json',
                            auth=FELIX_AUTH, timeout=8)
        if resp.status_code != 200:
            return jsonify({'status': 'down', 'console_status': resp.status_code}), 503

        data = resp.json()
        bundles = data.get('data', [])
        total = len(bundles)
        inactive = [b for b in bundles
                    if b.get('state') != 'Active'
                    and not (b.get('fragment') and b.get('state') == 'Resolved')]

        ok = len(inactive) == 0
        return jsonify({
            'status': 'ok' if ok else 'degraded',
            'total_bundles': total,
            'inactive_bundles': len(inactive),
            'problem_bundles': [b.get('symbolicName') for b in inactive[:10]],
        }), 200 if ok else 503
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=3013)

Step 2: Configure Vigilmon HTTP Monitor for Felix

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your Felix health endpoint: https://your-app.example.com/health/felix
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 5000ms
6. Under Alert channels, assign your Slack or PagerDuty channel
7. Save the monitor

Add a second monitor for the Felix Web Console itself (to detect if the admin interface is down):

• URL: https://your-app.example.com/system/console/bundles.json (with HTTP Basic auth header if your monitor supports it)
• Expected: 200
• Interval: 5 minutes
• Alert channel: P2 email — console unavailability affects operations but not end users directly

Vigilmon's multi-region probes distinguish a single-region network partition from a genuine Felix container failure.

Step 3: Heartbeat Monitoring for OSGi Service Availability

Bundle state monitoring catches bundles stuck in INSTALLED or RESOLVED, but not services that registered successfully and then threw an exception on first invocation, or DS components that deactivated due to a satisfied reference becoming unsatisfied mid-runtime.

Vigilmon heartbeat monitors catch these: your application code pings Vigilmon after each successful service invocation batch.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: felix-service-registry
3. Set the expected interval: 5 minutes
4. Set the grace period: 15 minutes
5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into a Declarative Services Component

// OrderProcessorComponent.java
import org.osgi.service.component.annotations.*;
import java.net.http.*;
import java.net.URI;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;

@Component(service = OrderProcessorComponent.class,
           immediate = true,
           property = "scheduler.expression=0/5 * * * * ?")
public class OrderProcessorComponent implements Runnable {

    @Reference
    private InventoryService inventoryService;

    @Reference
    private PaymentService paymentService;

    private final HttpClient http = HttpClient.newHttpClient();
    private final String heartbeatUrl = System.getenv("VIGILMON_HEARTBEAT_URL");
    private final AtomicLong lastPingMs = new AtomicLong(0);
    private static final long PING_INTERVAL_MS = 60_000;

    @Override
    public void run() {
        try {
            // Use OSGi services — if either was deactivated, this throws
            boolean inventoryOk = inventoryService.ping();
            boolean paymentOk = paymentService.ping();

            if (inventoryOk && paymentOk) {
                long now = System.currentTimeMillis();
                if (now - lastPingMs.get() > PING_INTERVAL_MS) {
                    lastPingMs.set(now);
                    pingHeartbeat();
                }
            }
        } catch (Exception e) {
            // Service deactivated or threw — do NOT ping; let heartbeat lapse
        }
    }

    private void pingHeartbeat() {
        if (heartbeatUrl == null) return;
        try {
            http.sendAsync(
                HttpRequest.newBuilder(URI.create(heartbeatUrl)).GET().build(),
                HttpResponse.BodyHandlers.discarding()
            );
        } catch (Exception ignored) {}
    }
}

BundleActivator Heartbeat (For Simple Felix Applications)

// VigilmonActivator.java
import org.osgi.framework.*;
import java.net.http.*;
import java.net.URI;
import java.util.concurrent.*;

public class VigilmonActivator implements BundleActivator {

    private ScheduledExecutorService scheduler;
    private final HttpClient http = HttpClient.newHttpClient();

    @Override
    public void start(BundleContext ctx) {
        String heartbeatUrl = ctx.getProperty("vigilmon.heartbeat.url");
        if (heartbeatUrl == null) return;

        scheduler = Executors.newSingleThreadScheduledExecutor();
        scheduler.scheduleAtFixedRate(() -> {
            try {
                // Check that the critical service is registered and available
                ServiceReference<?> ref = ctx.getServiceReference("com.example.CriticalService");
                if (ref != null && ctx.getService(ref) != null) {
                    http.sendAsync(
                        HttpRequest.newBuilder(URI.create(heartbeatUrl)).GET().build(),
                        HttpResponse.BodyHandlers.discarding()
                    );
                    ctx.ungetService(ref);
                }
            } catch (Exception ignored) {}
        }, 0, 5, TimeUnit.MINUTES);
    }

    @Override
    public void stop(BundleContext ctx) {
        if (scheduler != null) scheduler.shutdownNow();
    }
}

Step 4: Alert Routing for Felix Failures

Felix failures range from a single optional bundle failing to start (minor degradation) to a core bundle like org.apache.felix.scr failing (all DS components inactive). Route alerts by severity:

| Monitor | Alert Channel | Priority | |---|---|---| | HTTP: /health/felix | Slack + PagerDuty | P1 | | HTTP: Felix Web Console | Slack + email | P2 | | Heartbeat: service invocation success | Slack + PagerDuty | P1 | | Heartbeat: scheduled bundle tasks | Slack + email | P2 |

Set response time thresholds as early warning signals:

• Alert at 3000ms for the Felix health endpoint (slow health checks may indicate a bundle stuck in STARTING)
• Alert at 10000ms for the Web Console endpoint (Karaf/Felix console slowness often precedes bundle resolution failures)
• Alert at 10 minutes gap in service invocation heartbeat

For production Felix deployments running inside Karaf or Adobe AEM, set up a status page in Vigilmon showing bundle health, service registry state, and application-level heartbeat — this is the fastest path to root-cause analysis when a downstream service reports errors from an OSGi-powered application.

Summary

Felix's dynamic bundle model means services can silently disappear at runtime without any HTTP error or JVM crash. External monitoring catches bundle lifecycle failures and service registry gaps before they reach users:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/felix | Bundle state — inactive/unresolved bundles | | HTTP monitor on Web Console | Felix container liveness and admin interface | | Heartbeat monitor (service invocation) | OSGi service availability and DS component activation | | Heartbeat monitor (scheduled bundles) | Timer-driven OSGi bundle liveness |

Get started free at vigilmon.online — your first Felix bundle health monitor is running in under two minutes.