Using Prometheus Blackbox Exporter for External Endpoint Monitoring
A service can look healthy from inside its own network while customers receive timeouts, certificate warnings, or a blank response. Prometheus Blackbox Exporter helps detect that gap by testing endpoints from outside the application process. It can check HTTP and HTTPS services, DNS resolution, TCP ports, ICMP reachability, and other network behaviours.
This makes it useful for infrastructure teams that need to monitor the customer experience rather than simply confirm that a server process is running. A web server may have an active systemd unit and plenty of spare CPU while an expired TLS certificate or broken load-balancer route prevents anyone from using it.
The distinction matters in Australia, where geography creates meaningful differences in network performance. A site hosted in Sydney may respond quickly from Brisbane but behave differently for users in Perth, Adelaide, or a regional town connected through a busy NBN service. A single internal probe cannot represent all of those paths.
Blackbox Exporter is also a good fit for small operations. It works with the Prometheus and Alertmanager ecosystem, can run beside an existing monitoring stack, and avoids installing an agent on every external service. With sensible probe definitions and alert thresholds, it gives an operations team a practical early-warning system.
What Blackbox Exporter Actually Checks
Blackbox Exporter receives a target and a module name, performs the requested test, and exposes the result as Prometheus metrics. The exporter does not scrape the target’s application metrics. Instead, it behaves like an outside client and reports whether the expected network transaction succeeded.
For an HTTPS endpoint, useful measurements include probe_success, response time, returned status code, content length, redirect behaviour, and certificate information. DNS modules can verify resolution through a selected resolver, while TCP modules can confirm that a port accepts connections. ICMP can show basic reachability, although it is often blocked by firewalls and should not be treated as proof that an application is healthy.
This separation is valuable during incidents. A local application metric might show that PHP workers are available, while Blackbox Exporter reports that the public endpoint is returning HTTP 503. The two views together help identify whether the failure is in the application, reverse proxy, firewall, DNS, or a provider upstream.
Building a Useful Probe Configuration
The exporter’s blackbox.yml file defines modules. An HTTP module might enable redirects, require a 2xx response, verify TLS, and set a timeout. A separate module can check an internal health path, while another tests a full public page. Keeping these checks separate prevents a slow but valid content page from being confused with a fast service-readiness endpoint.
A basic configuration can look like this:
modules:
https_200:
prober: http
timeout: 10s
http:
method: GET
follow_redirects: true
preferred_ip_protocol: ip4
valid_status_codes: [200]
fail_if_ssl: false
fail_if_not_ssl: false
tcp_postgres:
prober: tcp
timeout: 5s
Prometheus then calls the exporter’s /probe endpoint and passes the real target as a parameter. Relabelling is the important part: the original target becomes the instance label, while Prometheus scrapes the exporter itself. Without that transformation, every metric can appear to belong to the exporter rather than the endpoint being tested.
Avoid placing credentials, session tokens, or private URLs in labels. Labels become part of the time-series identity and may be retained for a long time. Public endpoints are usually straightforward, but internal probes deserve the same care as any other monitoring configuration.
Designing Checks for Australian Networks
A single probe location can hide a regional outage. For an organisation serving customers around Australia, consider running exporters in at least two facilities, such as Sydney and Melbourne, with another vantage point in Brisbane or Perth when the service is commercially important. Cloud regions, office networks, and small VPS instances can all provide useful perspectives, provided their internet routes are documented.
Latency expectations should reflect the service’s real audience. A 250-millisecond response from Perth to a Sydney-hosted platform may be normal for a particular architecture, while the same value could indicate a fault for a Melbourne-based application. Alerting on absolute response time without understanding the route creates noisy pages and a fair bit of “what’s going on?” during the arvo.
Australian providers also have different peering and transit arrangements. A service can be reachable through one carrier while failing through another, and a local NBN fault may affect a branch office without affecting a cloud probe. Where customer impact is significant, combine cloud probes with a lightweight probe from a representative office or managed network.
For a public content or gaming property, an external HTTP check can confirm that the service is reachable from outside its hosting environment; a Dutch casino service is the kind of public-facing endpoint that benefits from checking status, TLS, redirects, and page availability separately.
Turning Metrics Into Actionable Alerts
The most basic alert is based on probe_success. A rule that fires when the value remains zero for several minutes can identify a genuine outage without paging for one dropped packet. The for duration should match the service’s importance and normal network behaviour. A payment endpoint may justify a shorter delay than a low-priority information page.
Availability alone is insufficient. Track probe_http_status_code for application failures, probe_duration_seconds for rising latency, and certificate metrics for impending TLS expiry. A service returning HTTP 200 with an empty error page may still pass a shallow probe, so content matching or a dedicated health endpoint can provide a stronger signal.
Example alert logic might include:
groups:
- name: external-endpoints
rules:
- alert: ExternalEndpointDown
expr: probe_success == 0
for: 5m
labels:
severity: critical
annotations:
summary: "External endpoint is unavailable"
- alert: ExternalEndpointSlow
expr: probe_duration_seconds > 3
for: 10m
labels:
severity: warning
Use Alertmanager routing to send critical failures to the on-call channel and slower performance warnings to a ticket or operations room. A good alert says which endpoint failed, from which probe location, and what was observed. “Blackbox failure” is much less useful than “Sydney HTTPS probe received 502 for 10 minutes”.
Security, Scaling, And Operations
Blackbox Exporter should not be exposed as an unrestricted public service. Restrict access to its metrics and probe endpoints with firewall rules, network policies, authentication, or a reverse proxy. Otherwise, an attacker may abuse it to make requests to arbitrary internal addresses, turning the exporter into a server-side request forgery tool.
Keep probe targets controlled through reviewed Prometheus configuration or service discovery. DNS, TCP, and HTTP checks can create unexpected load if a broad discovery rule includes thousands of endpoints. Use sensible scrape intervals, usually between 30 seconds and several minutes, and remember that every target creates work at each interval.
Container deployments are convenient, but ICMP probing may require extra Linux capabilities. Running as a non-root user is preferable where possible, with only the permissions required by the selected modules. For most web checks, HTTP and HTTPS probes need no special privileges.
Treat the exporter as production infrastructure. Monitor its own process, configuration reloads, scrape duration, and target count. A monitoring system that silently stops probing is a false sense of security. Store configuration in version control, test module changes before rollout, and record which vantage points represent which customer populations.
Using Results During Incident Response
External probe metrics are most powerful when correlated with logs, traces, load-balancer statistics, and provider status information. If all locations fail at the same time, investigate the application, DNS, certificate, or hosting platform. If only one location fails, examine routing, peering, regional filtering, or the probe host itself.
A timeline helps separate symptoms from causes. Compare the first failed probe with deployment events, DNS changes, certificate renewals, firewall updates, and upstream alerts. The same approach is useful for market-facing services: operational assumptions should be checked against real customer access patterns, which is a theme often reflected in broader SMB research about how smaller organisations evaluate technology and business services.
Synthetic monitoring should also cover the path customers actually use. A homepage check is a useful starting point, but login, API, checkout, booking, and status endpoints may fail independently. Use safe test accounts or purpose-built health routes, and avoid transactions that create real orders or financial activity.
Review the probes after each incident. If an alert was technically correct but operationally confusing, improve its labels and runbook. If customers experienced a failure without an alert, add a check at the missing layer. Over time, the probe catalogue becomes a map of the service’s external dependencies.
Start with a handful of critical HTTPS endpoints, deploy probes from the locations that match your Australian users, and add alerts for availability, latency, status codes, and certificate expiry. Prometheus Blackbox Exporter is small enough to introduce quickly, yet flexible enough to become a dependable part of a broader observability platform.
Karl Katzke