#Configuring the Collector Metrics

The OpenTelemetry Collector exposes internal metrics that provide valuable insights into its performance and operational health. These metrics enable you to monitor the Collector's behavior and troubleshoot potential issues effectively.

#Overview

The Collector provides metrics across several key areas:

• Collector memory usage
• CPU utilization
• Number of active traces and spans processed
• Dropped spans, logs, or metrics
• Exporter and receiver statistics

When you deploy a Collector instance, the Operator provisions a dedicated monitoring service (<instance_name>-collector-monitoring) that serves these internal metrics on port 8888. You can point Prometheus or any compatible scraping tool at this service to collect the metrics.

#Enabling Collector Metrics

Procedure

Enable metrics collection for a Collector instance by setting spec.observability.metrics.enableMetrics to true in the CR:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: <name>
  labels:
    prometheus: kube-prometheus
spec:
  observability:
    metrics:
      enableMetrics: true
  config:
    service:
      telemetry:
        metrics:
          readers:
          - pull:
              exporter:
                prometheus:
                  host: 0.0.0.0
                  port: 8888

Verification

You can use the Prometheus web console to verify successful configuration:

1. Access the Prometheus web console in your Alauda Container Platform cluster.

2. Navigate to Status → Targets.

3. Check that the ServiceMonitors or PodMonitors in the <instance_name>-collector format have the Up status.

#Identifying the Collector that produced a metric

The Collector no longer stamps service_name, service_instance_id, and service_version onto every internal metric series. These attributes are exposed only through the target_info metric, which is the conventional Prometheus and OpenTelemetry representation of resource identity.

To attribute a metric to a specific Collector, join the metric against target_info on the shared job and instance labels:

otelcol_receiver_accepted_spans
  * on (job, instance) group_left ("service.name", "service.version")
  target_info{endpoint="monitoring"}

Two details of this query are not optional. Omitting either one produces a broken result, so both are explained below.

#Always restrict target_info to the monitoring endpoint

target_info is not specific to the Collector's internal telemetry. Every scrape target that carries OpenTelemetry resource attributes publishes a metric of that name, and the Collector's own Prometheus exporter port (8889 in the Configuration Example) publishes one target_info series per upstream application it receives data from. All of those series come from a single scrape target, so they share one job and instance pair:

found duplicate series for the match group {instance="10.3.6.73:8889", job="my-collector"}
on the right hand-side of the operation: [...];
many-to-many matching not allowed: matching labels must be unique on one side

Prometheus checks that the group_left right-hand side is unique before it matches anything against the left-hand side. The otelcol_* metrics exist only on port 8888 and never collide with these series, but the query still fails, because the duplicates are present in the right-hand side vector at all.

Restricting the selector to the monitoring endpoint removes them. The Operator names the internal telemetry port monitoring, which Prometheus Operator copies into the endpoint label, so target_info{endpoint="monitoring"} selects exactly one series per Collector. If your Prometheus does not set an endpoint label, match on the job name of the monitoring service instead, which the Operator derives from the Collector name:

target_info{job=~".*-collector-monitoring"}

#Confirm how your Prometheus spells the attribute names

The Collector exposes the resource attributes under their original OpenTelemetry names — service.name, service.instance.id, and service.version. Whether the dots survive into storage depends on the Prometheus version:

Query target_info{endpoint="monitoring"} on its own and read the label names off the result before writing the join. The examples on this page use the dotted names, which is what Alauda Container Platform stores.

#Reusing the join across dashboards

If rewriting every query is not practical, define a recording rule that performs the join once and query the recorded series instead:

groups:
  - name: otelcol-internal-metrics
    rules:
      - record: otelcol:receiver_accepted_spans:with_identity
        expr: |
          otelcol_receiver_accepted_spans
            * on (job, instance) group_left ("service.name", "service.version")
            target_info{endpoint="monitoring"}