Custom Metrics Exporter Development for Prometheus

Custom Prometheus exporter: where metrics collection breaks

Typical picture: services respond, dashboard is green, and we learn about an incident from a user complaint. Formally, metrics collection is configured, but it covers only components that expose metrics themselves. Everything else does not exist for observability—although degradation most often begins there.

Metrics monitoring gaps stem from three sources. Some data lives in closed systems—proprietary or third-party—where you cannot install a standard agent and they have no open endpoint.

Some data is provided in an inconvenient form: numbers exist, but are in logs, tables, or internal API responses, from where nobody collects them.

Another part appears only during manual reconciliation—an engineer compiles reports and looks for discrepancies by eye that no one sees in real time.

A standard exporter does not help here: it needs a source that already speaks the language of metrics.

If a service exposes only lines in a log, aggregates in a database, or counters in a queue, between it and the monitoring system you need a translation layer—with its own polling logic, response parsing, and bringing values to a common form.

A custom Prometheus exporter turns such sources into regular metrics of closed systems: they are visible on common graphs, and alerts and aggregation rules fire on them.

Next, we break down how this is addressed—from selecting sources and polling design to verifying that the numbers on the graph do not diverge from reality.

What a custom metrics collector provides: effect in measurable values

Incident diagnosis time

Application metrics land on the same time scale as host, queue, and database metrics, so there is no longer a need to manually correlate logs from different systems.

The cause of degradation can be read from the graph: you can see at which step latency grew or the success rate dropped. The on-call engineer goes to a specific service instead of checking the entire chain.

Early failure detection

The collector emits signals for internal queues, connection pools, and request retries—things that previously showed up only as a user complaint. Rules fire on deviation from the normal load profile long before the service stops responding.

Completeness of the monitoring picture

Standard agents see the process but not the meaning: how many orders were processed, how many operations were rejected. Custom metrics close this gap, and the dashboard reflects the entire request path—from the ingress gateway to the write to storage.

Share of failures before users contact support

When signals are based on business events, not just availability, most failures are detected by the on-call engineer rather than the support team. The "everything is broken" request stops being the primary detection channel.

Common point of truth

One set of metrics is used by on-call engineers, developers, and the product owner, without reconciling data from multiple dashboards. The argument "it worked on our side" is settled by objective numbers.

Exporter implementation options: comparison of approaches

The choice of design is determined by three things: whether there is access to the service code, how long the process lives, and what happens to metrics when the collector is unavailable. Below are four options that are realistically chosen between, and their limitations.

In practice, the design is almost always combined: permanent services expose metrics themselves, external systems are polled by a separate collector, and short jobs go through a gateway.

We separate these contours by different collection rules so that a failure of one does not corrupt the series of the others, and we keep the transition between options reversible—the exporter can be moved from the application to a separate service without rewriting queries and dashboards.

How we implement metrics collection: work stages

Metrics monitoring implementation stages go from inventory to handover to operations, and each relies on the result of the previous one.

The order removes the main cause of failures: a custom Prometheus exporter is written before it is determined that some metrics are exposed by default and some require changes on the source side.

1. Audit of metric sources. Inventory of systems: applications, databases, queues, load balancers, external services. For each, we record what is exposed by default and what we will have to calculate ourselves, and at what frequency.
2. Flow design and data model. Metric types (counter, gauge, histogram), label set and cardinality limit so storage does not grow uncontrollably. We decide where a cache is needed and where direct polling is needed.
3. Implementation of collection. Parsing source responses, resilience to delays and partial failures, a service health endpoint and a self-monitoring metric. We perform polling read-only.
4. Data collection configuration. Targets, polling intervals, recalculation rules, alert thresholds, and monitoring dashboards. A metric without an alert and a graph is useless.
5. Load testing. We compare values with primary data, examine divergence, behavior when the source is unavailable, and cardinality growth. Load on the source remains within agreed limits.
6. Handover to operations. Configurations, flow designs, on-call policy, and instructions on how to add a new metric on your own without rewriting the collector.

What remains in the infrastructure after implementation

Implementation ends not when metrics start flowing, but when the on-call shift can read them without the author and can fix collection on their own. Therefore handover is a fixed list of artifacts, not a verbal story on a call. Below is what remains in your hands after delivery of a custom Prometheus exporter.

• Collection configurations: exporter manifests, targets and polling intervals, label set—with an explanation of what changes when a new node is added.
• Metric flow design: the path from source to storage and alerting rules, with marked points of failure and ways to work around them.
• Metric documentation: type, units, source, normal range, and what deviation means—using this description, the metric is interpreted without the author.
• Metric access rules: who reads, who changes alert thresholds, how access is granted and revoked, where keys are stored.
• Collection policy: polling frequency, series retention period, stale data cleanup policy.
• Response policy: which alerts are critical, whom they are addressed to, what is considered false, and how the threshold is adjusted.
• Extension instructions: how to add a new metric, test it locally, and roll it out without breaking already collected ones.
• Operations handover protocol: demonstration to the on-call shift, acceptance checklist, support window, and the point after which changes follow the normal process.

Case: billing metrics without manual reconciliation

Before implementation, charges were calculated in the billing service, events went to a queue, and nightly aggregates were stored in storage—three sources that no one compared line by line.

Discrepancies surfaced only from complaints: a client noticed an extra charge, support opened an investigation, and the search took up to three weeks. By that time, the invoice had been issued and paid.

What we did

We wrote a custom exporter that subscribes to the stream of billing events and reads the tariff directory without interfering with billing logic—read-only.

Each event becomes a counter with labels by client, tariff, and operation type, and the discrepancy between the sum by events and the sum in the invoice goes out as a separate metric. Prometheus scrapes the data, and a threshold alert rule fires on deviation for a specific client.

• Stages: charge → invoice → write-off, with a gap at each transition.
• Integrity: number of events in the queue versus number processed.
• Reconciliation with the previous period—for slowly accumulating discrepancies.
• Alerts split into critical and those requiring investigation.

Result

The discrepancy is visible at the moment it occurs: the signal arrives a few minutes after the deviation instead of three weeks later.

The entire billing stream is covered by reconciliation—previously only selected invoices were checked based on complaints, now no operation is missed.

The load on billing did not increase: the exporter only reads the stream and does not participate in processing payments. The collector survives restarts—the position in the stream is preserved, and no reprocessing or double counting of metrics occurs.

How a custom exporter differs from off-the-shelf collectors

An off-the-shelf exporter handles the task completely as long as the metric source is standard. Node state, database, queue, or network service availability are described in the official set—all that remains is to install the collector and set addresses with a polling interval.

We write no code: there is nothing to maintain, and behavior is predictable for any engineer on the team.

Limitations begin where data lies inside your logic. Prometheus only scrapes a metrics endpoint over the network—it does not know how to retrieve balances from an accounting system, license status from a vendor, or a queue of unprocessed requests.

The workaround—a scheduled script with a text file—lives only until the first failure: the metric goes stale, is duplicated on restart, and when data is absent it returns zero instead of a gap.

The second limitation is the data shape. Off-the-shelf collectors expose raw counters, while decisions are made from derived values: error rate, wait time, number of objects in an abnormal state.

A custom collector calculates this at the collection point, keeps the last known value, marks stale series, and retries the request with increasing delay.

Four questions decide the choice: does the source have a documented metrics endpoint; can the data model be expressed through configuration; does a full traversal of objects fit within the polling interval; are access separation and per-client filtering needed.

A negative answer to at least one—and the off-the-shelf solution will have to be padded with workarounds.

A hybrid is more beneficial: we collect standard things with off-the-shelf collectors, and write our own for the gap—a compact service with tests, a separate failure contour, and its own health metrics.

Technical concerns about metrics collection and answers

A custom exporter sits between the monitoring system and the production contour, so objections about load, access, and further support are valid. Below we answer those most often raised at the evaluation stage. All four are addressed at the level of collection architecture, not promises.

How will you get access to a closed system?

We install the exporter inside your contour: it opens only an outgoing connection to the collection server, and inbound ports are not forwarded.

The account has read-only rights to the required views; for isolated contours, transfer goes over a secure channel already approved by you.

Can a metric be added without stopping the service?

The metric set is described declaratively: a new one is added by editing the configuration and restarting only the exporter; the target service is not stopped. We fix the naming and label scheme in advance so that additions do not break dashboards and alerting rules.

What is included in metrics collection support?

We hand over configurations, the metric scheme, and instructions for typical changes—your team adds metrics without us. We separately describe behavior when the source is unavailable, so that a collection failure can be seen separately from a failure of the service itself.

Let's discuss your metrics collection task

So that the estimate of metrics collection work is substantive rather than "based on feelings," in the first conversation we need a picture of your infrastructure: which sources already expose data, what is collected manually, where there are discrepancies with reality.

We figure out where a custom Prometheus exporter is truly needed and where standard tools and configuration are sufficient.

What we clarify before returning with a metrics collection implementation plan:

• Which metric sources exist now and in what form they expose data—built-in export, log files, internal APIs, database queries.
• What exactly should become measurable: request processing latency, queue depth, error rate, connection state, data freshness.
• What polling frequency and retention depth to base calculations on so that data is not lost and the registry does not grow uncontrollably.
• What labeling rules are needed so that metrics from different services can be compared with each other and do not turn into thousands of unnecessary label sets.
• How data gets into your dashboards and alerts and which signals should wake the on-call engineer at night and which should only be written to a report.
• Requirements for access, network availability, and the procedure for stopping collection if the exporter has to be removed or switched to a backup source.

Describe your monitoring task—we will return with an estimate of metrics collection work and an implementation plan: stages, configuration composition, timelines, and a way to verify that metrics correspond to the real state of the system.