Docs » µAPM Getting Started » Deploy an OpenTelemetry Collector for SignalFx µAPM

Deploy an OpenTelemetry Collector for SignalFx µAPM 🔗

Important

The original µAPM product, released in 2019, is now called µAPM Previous Generation (µAPM PG). In the documentation, µAPM now refers to the product released on March 31, 2020.

If you are using µAPM Previous Generation (µAPM PG), see Deploying the SignalFx Smart Gateway.

After installing the SignalFx Smart Agent on each instance, you can optionally deploy the OpenTelemetry Collector in each datacenter/region/cluster where traced applications run. In general, the OpenTelemetry Collector should receive data from the SignalFx Smart Agent.

Sending trace spans through the OpenTelemetry Collector gives the OpenTelemetry Collector an opportunity to modify span attributes prior to being exported to the configured backend. In addition, the collector reduces the number of devices that require internet access. For more information, see Attribute Processor on GitHub.

For more information about deploying an OpenTelemetry Collector, see the OpenTelemetry docs.

After you deploy an OpenTelemetry Collector, configure a SignalFx Smart Agent to forward data to it over the SAPM protocol. You can deploy multiple OpenTelemetry Collectors behind a load balancer for a high-availability configuration. In this deployment model, configure the Smart Agent to point to an OpenTelemetry Collector VIP.

Important

µAPM requires OpenTelemetry Collector version 0.3.0 or above.

How the OpenTelemetry Collector works 🔗

The OpenTelemetry Collector uses pipelines to receive, process, and export trace data with components conveniently known as receivers, processors, and exporters. Set up pipelines with services. You can also add extensions that provide an OpenTelemetry Collector with additional functionality, such as diagnostics and health checks. The OpenTelemetry Collector has two versions: a core version and a contributions version. The core version provides receivers, processors, and exporters for general use. The contributions version provides receivers, processors, and exporters for specific vendors and use cases.

SignalFx uses these contributions versions for receivers and exporters to send data to an OpenTelemetry Collector and to receive data from an OpenTelemetry Collector:

Component Name Description
Receiver sapm Component that sets the endpoint for receiving trace data with the SAPM format.
Receiver signalfx Component that sets the endpoint for receiving metrics data with the SignalFx metric data format.
Exporter sapm Component that forwards data to SignalFx with the SAPM format.
Exporter signalfx Component that forwards data to SignalFx with the metric data format.

OpenTelemetry provides information about some Collector components on GitHub:

For more information about each component, including services, see Configuration on the OpenTelemetry site.

Plan your OpenTelemetry Collector deployment 🔗

The OpenTelemetry Collector can be scaled up or out as needed. A single Collector is generally capable of over 10,000 spans per second per CPU core. Try to leverage a ratio of 1:2 for CPU:memory and to allocate at least a CPU core per Collector. You can also deploy multiple Collectors. Each Collector runs independently, so sizing increases linearly with the number of Collectors you deploy.

Deploy an OpenTelemetry Collector from a binary or as a Docker image 🔗

Follow these steps to deploy an OpenTelemetry Collector from a binary or as a Docker image. Use the contributions version of the OpenTelemetry Collector.

  1. Download the latest release of the OpenTelemetry Collector contributions version on GitHub.

  2. Create a configuration YAML file. These steps use collector.yaml for the example file name. You have to include at least the following options in the file:
    extensions:
      health_check:
      zpages:
        endpoint: 0.0.0.0:55679
    receivers:
      otlp:
        endpoint: 0.0.0.0:55680
      sapm:
        endpoint: 0.0.0.0:7276
      signalfx:
        endpoint: 0.0.0.0:9943
      # This section is used to collect OpenTelemetry metrics
      # Even if just a SignalFx µAPM customer, these metrics are included
      prometheus:
        config:
          scrape_configs:
            - job_name: 'otel-collector'
              scrape_interval: 10s
              static_configs:
                - targets: ['localhost:8888']
                  # If you want to use the environment filter
                  # In the SignalFx dashboard
                  #labels:
                    #environment: demo
              metric_relabel_configs:
                - source_labels: [ __name__ ]
                  regex: '.*grpc_io.*'
                  action: drop
    processors:
      batch:
      # Optional: If you have multiple environments
      # attributes/copyfromexistingkey:
        # actions:
        # - key: environment
          # from_attribute: YOUR_EXISTING_TAG
          # action: upsert
      # attributes/newenvironment:
        # actions:
        # - key: environment
          # value: "YOUR_ENVIRONMENT_NAME"
          # action: insert
      queued_retry:
      memory_limiter:
        ballast_size_mib: 683
        limit_mib: 1800
        spike_limit_mib: 500
    exporters:
      # Traces
      sapm:
        access_token: "YOUR_ACCESS_TOKEN"
        endpoint: "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/trace"
      # Metrics
      signalfx:
        access_token: "YOUR_ACCESS_TOKEN"
        realm: "YOUR_SIGNALFX_REALM"
    service:
      pipelines:
      traces:
        receivers: [sapm, otlp]
        processors: [memory_limiter, batch, queued_retry]
        exporters: [sapm]
        metrics:
          receivers: [signalfx, prometheus, otlp]
          exporters: [signalfx]
      extensions: [health_check, zpages]
    
  3. Deploy an Open Telemetry Collector.

    Deploy from a Docker container:

    $ docker run -p 7276:7276 -p 8888:8888 -p 9943:9943 -p 55679:55679 -p 55680:55680 \
        -v collector.yaml:/etc/collector.yaml:ro \
        --name otelcontribcol otel/opentelemetry-collector-contrib:latest \
            --config /etc/collector.yaml --new-metrics --legacy-metrics=false \
    

    Deploy from a binary:

    $ otelcontribcol --config collector.yaml --new-metrics --legacy-metrics=false
    

Deploy an OpenTelemetry Collector in Kubernetes 🔗

To deploy the OpenTelemetry Collector in Kubernetes, use a configuration YAML file like this:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  labels:
    app: opentelemetry
    component: otel-collector-conf
data:
  otel-collector-config: |
    receivers:
      otlp:
        endpoint: 0.0.0.0:55680
      sapm:
        endpoint: 0.0.0.0:7276
      signalfx:
        endpoint: 0.0.0.0:9943
      prometheus:
        config:
          scrape_configs:
            - job_name: 'otel-collector'
              scrape_interval: 10s
              static_configs:
                - targets: ['localhost:8888']
                  # If you want to use the environment filter
                  # In the SignalFx dashboard
                  #labels:
                    #environment: demo
              metric_relabel_configs:
                - source_labels: [ __name__ ]
                  regex: '.*grpc_io.*'
                  action: drop
    processors:
      queued_retry:
      batch:
      memory_limiter:
        ballast_size_mib: 683
        limit_mib: 1800
        spike_limit_mib: 500
    extensions:
      health_check: {}
      zpages:
        endpoint: 0.0.0.0:55679
    exporters:
      sapm:
        access_token: "YOUR_ACCESS_TOKEN"
        endpoint: "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/trace"
      signalfx:
        access_token: "YOUR_ACCESS_TOKEN"
        realm: "YOUR_SIGNALFX_REALM"
    service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [sapm, otlp]
          processors: [memory_limiter, batch, queued_retry]
          exporters: [sapm]
        metrics:
          receivers: [signalfx, prometheus, otlp]
          exporters: [signalfx]
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  labels:
    app: opentelemetry
    component: otel-collector
spec:
  ports:
  - name: signalfx # Default endpoint for infrastructure monitoring.
    port: 9943
  - name: otlp # Default endpoint for OpenTelemetry receiver.
    port: 55680
    protocol: TCP
    targetPort: 55680
  - name: metrics # Default endpoint for querying metrics.
    port: 8888
  - name: sapm # Default endpoint for SAPM traces.
    port: 7276
  - name: zpages # Default endpoint for zpages.
    port: 55679
  selector:
    component: otel-collector
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  labels:
    app: opentelemetry
    component: otel-collector
spec:
  selector:
    matchLabels:
      app: opentelemetry
      component: otel-collector
  minReadySeconds: 5
  progressDeadlineSeconds: 120
  replicas: 1 #TODO - adjust this to your own requirements
  template:
    metadata:
      labels:
        app: opentelemetry
        component: otel-collector
    spec:
      containers:
      - command:
          - "/otelcontribcol"
          - "--config=/conf/otel-collector-config.yaml"
            # Memory Ballast size should be max 1/3 to 1/2 of memory.
          - "--mem-ballast-size-mib=683"
          - "--new-metrics"
          - "--legacy-metrics=false"
        image: otel/opentelemetry-collector-contrib:latest
        name: otel-collector
        resources:
          limits:
            cpu: 1
            memory: 2Gi
          requests:
            cpu: 200m
            memory: 400Mi
        ports:
        - containerPort: 55680 # Default endpoint for OpenTelemetry receiver.
        - containerPort: 9411 # Default endpoint for Zipkin receiver.
        - containerPort: 8888  # Default endpoint for querying metrics.
        - containerPort: 7276 # Default endpoint querying SAPM receiver.
        - containerPort: 55679 # Default endpoint for zpages.
        volumeMounts:
        - name: otel-collector-config-vol
          mountPath: /conf
          # - name: otel-collector-secrets
            # mountPath: /secrets
        livenessProbe:
          httpGet:
            path: /
            port: 13133 # Health Check extension default port.
        readinessProbe:
          httpGet:
            path: /
            port: 13133 # Health Check extension default port.
      volumes:
        - configMap:
            name: otel-collector-conf
            items:
              - key: otel-collector-config
                path: otel-collector-config.yaml
          name: otel-collector-config-vol
          # - secret:
            # name: otel-collector-secrets
              # items:
                # - key: cert.pem
                  # path: cert.pem
                # - key: key.pem
                  # path: key.pem