Using Prometheus Blackbox Exporter for External Endpoint Monitoring

A service can look healthy from inside its own network while customers receive timeouts, certificate warnings, or a blank response. Prometheus Blackbox Exporter helps detect that gap by testing endpoints from outside the application process. It can check HTTP and HTTPS services, DNS resolution, TCP ports, ICMP reachability, and other network behaviours.

This makes it useful for infrastructure teams that need to monitor the customer experience rather than simply confirm that a server process is running. A web server may have an active systemd unit and plenty of spare CPU while an expired TLS certificate or broken load-balancer route prevents anyone from using it.

The distinction matters in Australia, where geography creates meaningful differences in network performance. A site hosted in Sydney may respond quickly from Brisbane but behave differently for users in Perth, Adelaide, or a regional town connected through a busy NBN service. A single internal probe cannot represent all of those paths.

Blackbox Exporter is also a good fit for small operations. It works with the Prometheus and Alertmanager ecosystem, can run beside an existing monitoring stack, and avoids installing an agent on every external service. With sensible probe definitions and alert thresholds, it gives an operations team a practical early-warning system.

What Blackbox Exporter Actually Checks

Blackbox Exporter receives a target and a module name, performs the requested test, and exposes the result as Prometheus metrics. The exporter does not scrape the target’s application metrics. Instead, it behaves like an outside client and reports whether the expected network transaction succeeded.

For an HTTPS endpoint, useful measurements include probe_success, response time, returned status code, content length, redirect behaviour, and certificate information. DNS modules can verify resolution through a selected resolver, while TCP modules can confirm that a port accepts connections. ICMP can show basic reachability, although it is often blocked by firewalls and should not be treated as proof that an application is healthy.

This separation is valuable during incidents. A local application metric might show that PHP workers are available, while Blackbox Exporter reports that the public endpoint is returning HTTP 503. The two views together help identify whether the failure is in the application, reverse proxy, firewall, DNS, or a provider upstream.

Building a Useful Probe Configuration

The exporter’s blackbox.yml file defines modules. An HTTP module might enable redirects, require a 2xx response, verify TLS, and set a timeout. A separate module can check an internal health path, while another tests a full public page. Keeping these checks separate prevents a slow but valid content page from being confused with a fast service-readiness endpoint.

A basic configuration can look like this:

modules:
  https_200:
    prober: http
    timeout: 10s
    http:
      method: GET
      follow_redirects: true
      preferred_ip_protocol: ip4
      valid_status_codes: [200]
      fail_if_ssl: false
      fail_if_not_ssl: false

  tcp_postgres:
    prober: tcp
    timeout: 5s

Prometheus then calls the exporter’s /probe endpoint and passes the real target as a parameter. Relabelling is the important part: the original target becomes the instance label, while Prometheus scrapes the exporter itself. Without that transformation, every metric can appear to belong to the exporter rather than the endpoint being tested.

Avoid placing credentials, session tokens, or private URLs in labels. Labels become part of the time-series identity and may be retained for a long time. Public endpoints are usually straightforward, but internal probes deserve the same care as any other monitoring configuration.

Designing Checks for Australian Networks

A single probe location can hide a regional outage. For an organisation serving customers around Australia, consider running exporters in at least two facilities, such as Sydney and Melbourne, with another vantage point in Brisbane or Perth when the service is commercially important. Cloud regions, office networks, and small VPS instances can all provide useful perspectives, provided their internet routes are documented.

Latency expectations should reflect the service’s real audience. A 250-millisecond response from Perth to a Sydney-hosted platform may be normal for a particular architecture, while the same value could indicate a fault for a Melbourne-based application. Alerting on absolute response time without understanding the route creates noisy pages and a fair bit of “what’s going on?” during the arvo.

Australian providers also have different peering and transit arrangements. A service can be reachable through one carrier while failing through another, and a local NBN fault may affect a branch office without affecting a cloud probe. Where customer impact is significant, combine cloud probes with a lightweight probe from a representative office or managed network.

For a public content or gaming property, an external HTTP check can confirm that the service is reachable from outside its hosting environment; a Dutch casino service is the kind of public-facing endpoint that benefits from checking status, TLS, redirects, and page availability separately.

Turning Metrics Into Actionable Alerts

The most basic alert is based on probe_success. A rule that fires when the value remains zero for several minutes can identify a genuine outage without paging for one dropped packet. The for duration should match the service’s importance and normal network behaviour. A payment endpoint may justify a shorter delay than a low-priority information page.

Availability alone is insufficient. Track probe_http_status_code for application failures, probe_duration_seconds for rising latency, and certificate metrics for impending TLS expiry. A service returning HTTP 200 with an empty error page may still pass a shallow probe, so content matching or a dedicated health endpoint can provide a stronger signal.

Example alert logic might include:

groups:
  - name: external-endpoints
    rules:
      - alert: ExternalEndpointDown
        expr: probe_success == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "External endpoint is unavailable"

      - alert: ExternalEndpointSlow
        expr: probe_duration_seconds > 3
        for: 10m
        labels:
          severity: warning

Use Alertmanager routing to send critical failures to the on-call channel and slower performance warnings to a ticket or operations room. A good alert says which endpoint failed, from which probe location, and what was observed. “Blackbox failure” is much less useful than “Sydney HTTPS probe received 502 for 10 minutes”.

Security, Scaling, And Operations

Blackbox Exporter should not be exposed as an unrestricted public service. Restrict access to its metrics and probe endpoints with firewall rules, network policies, authentication, or a reverse proxy. Otherwise, an attacker may abuse it to make requests to arbitrary internal addresses, turning the exporter into a server-side request forgery tool.

Keep probe targets controlled through reviewed Prometheus configuration or service discovery. DNS, TCP, and HTTP checks can create unexpected load if a broad discovery rule includes thousands of endpoints. Use sensible scrape intervals, usually between 30 seconds and several minutes, and remember that every target creates work at each interval.

Container deployments are convenient, but ICMP probing may require extra Linux capabilities. Running as a non-root user is preferable where possible, with only the permissions required by the selected modules. For most web checks, HTTP and HTTPS probes need no special privileges.

Treat the exporter as production infrastructure. Monitor its own process, configuration reloads, scrape duration, and target count. A monitoring system that silently stops probing is a false sense of security. Store configuration in version control, test module changes before rollout, and record which vantage points represent which customer populations.

Using Results During Incident Response

External probe metrics are most powerful when correlated with logs, traces, load-balancer statistics, and provider status information. If all locations fail at the same time, investigate the application, DNS, certificate, or hosting platform. If only one location fails, examine routing, peering, regional filtering, or the probe host itself.

A timeline helps separate symptoms from causes. Compare the first failed probe with deployment events, DNS changes, certificate renewals, firewall updates, and upstream alerts. The same approach is useful for market-facing services: operational assumptions should be checked against real customer access patterns, which is a theme often reflected in broader SMB research about how smaller organisations evaluate technology and business services.

Synthetic monitoring should also cover the path customers actually use. A homepage check is a useful starting point, but login, API, checkout, booking, and status endpoints may fail independently. Use safe test accounts or purpose-built health routes, and avoid transactions that create real orders or financial activity.

Review the probes after each incident. If an alert was technically correct but operationally confusing, improve its labels and runbook. If customers experienced a failure without an alert, add a check at the missing layer. Over time, the probe catalogue becomes a map of the service’s external dependencies.

Start with a handful of critical HTTPS endpoints, deploy probes from the locations that match your Australian users, and add alerts for availability, latency, status codes, and certificate expiry. Prometheus Blackbox Exporter is small enough to introduce quickly, yet flexible enough to become a dependable part of a broader observability platform.

Experience

Information Technology Consulting

Independent Practice

Provides IT consulting services focused on infrastructure planning, cloud migration strategy, and systems architecture. Engagements draw on years of hands-on sysadmin and development experience across Linux, Windows, and hybrid environments.

K9 Search & Rescue Volunteer

Ongoing

Active participant in K9 Search & Rescue operations, combining technical logistics skills with field support for canine search teams.

Karl Katzke's Blog

October 2006 – May 2014

Published a long-running personal technology blog covering cloud vs. in-house infrastructure, F# and Mono on OSX, hardware vendor critiques, RAID card performance analysis, and sysadmin storytelling. Notable posts include "When Sysadmins Ruled the Earth" (May 15, 2014) and "Getting Started with F# and Mono on OSX" (December 22, 2012).

Credentials

A small badge icon with a shield shape in muted blue tones on a light background

Systems Administration

Deep experience with Linux (RHEL, SLES, CentOS), high-availability clusters, and STONITH configurations.

A small badge icon with a gear shape in muted blue tones on a light background

Cloud Infrastructure

Practical knowledge of AWS EC2, reserved instances, and cost analysis for cloud vs. on-premises deployments.

A small badge icon with a code symbol in muted blue tones on a light background

Development

Proficient in F#, PHP (Symfony), and cross-platform tooling including Mono and MonoDevelop on OSX.

Studies

F# & Functional Programming

Self-directed, 2012

Explored strongly typed functional programming with F# on OSX using the Mono runtime. Published a detailed getting-started guide covering toolchain setup and cross-platform game development research.

High-Availability & Cluster Management

Professional Development, 2009

Configured and documented crm_mon email alerting for STONITH events on SLES11-HAE clusters, integrating with Nagios monitoring for production environments.

Hardware & Storage Performance

Ongoing

Conducted hands-on benchmarking of SATA/SAS RAID controllers including HighPoint RocketRaid 2740 and LSI/SuperMicro AOC-USASLP2-H8iR, comparing against software RAID configurations.

Skills

A small icon representing a server with clean geometric lines in slate blue

Linux Administration

RHEL, SLES, CentOS — package management, kernel tuning, HA clustering, and monitoring integration.

A small icon representing a cloud shape with clean geometric lines in slate blue

Cloud Architecture

AWS EC2, reserved-instance planning, cost modeling, and hybrid infrastructure strategy.

A small icon representing code brackets with clean geometric lines in slate blue

F# & .NET/Mono

Functional programming on OSX, MonoDevelop toolchain, and cross-platform game-dev exploration.

A small icon representing a database cylinder with clean geometric lines in slate blue

PHP & Symfony

Web application development with the Symfony framework and the broader PHP ecosystem.

A small icon representing a storage drive with clean geometric lines in slate blue

Storage & RAID

SATA/SAS controller evaluation, md RAID configuration, and performance benchmarking.

A small icon representing a shield with clean geometric lines in slate blue

High Availability

Pacemaker, STONITH, crm_mon alerting, and Nagios integration for production cluster monitoring.