Docs » APM PG Instrumentation Overview

APM PG Instrumentation Overview 🔗

Back to Instrumentation Guide contents

Instrumentation is the process through which your applications’ code is extended to capture and report trace spans for the operations of interest in the path of handling a particular user request or transaction. Depending on which language(s) your applications are written in, you may benefit from automatic instrumentation (“auto-instrumentation”) capabilities. Additionally, or in lieu of, you can manually instrument the areas of your code and business logic that matter to you the most.

Automatic instrumentation 🔗

Automatic instrumentation, also known as auto-instrumentation, is the process through which a run-time automated process can automatically identify which frameworks and libraries are in use within your application and instrument those libraries to capture tracing instrumentation – all without having to make any code change.

SignalFx Microservices APM offers automatic instrumentation for the following programming languages:

You can extend the scope and coverage of the traces and spans captured by auto-instrumentation using manual, custom instrumentation. All auto-instrumentation techniques listed above will automatically provide a pre-configured, OpenTracing-compatible tracer that you can re-use in other parts of your application where you want to add instrumentation.


You should not attempt to use automatic instrumentation or instrumentation agents from multiple vendors at the same time. This will lead to unpredictable results, broken instrumentation, and in some cases might crash your application. To guarantee the stability of your applications and the best instrumentation experience, only use one vendor instrumentation or agent at a time.

Manual instrumentation 🔗

Manual, or custom, instrumentation, consists in modifying your application code to selectively capture trace spans and relevant span metadata tags around the blocks of code that you want to instrument, trace, and gain visibility on. This is typically a low-to-medium effort by software developers, akin to capturing custom metrics albeit with a different set of caveats and pitfalls to pay attention to.

To avoid vendor lock-in, including lock-in to a particular open-source tracing solution, we recommend that custom instrumentation be implemented using the OpenTracing APIs. OpenTracing is a CNCF project and is rapidly becoming one of the de-facto standards for code instrumentation, offering clean APIs in a growing spectrum of programming languages. We then recommend the use of the corresponding Jaeger tracer client library that implements the OpenTracing APIs and will be responsible for reporting spans to SignalFx.

For more information on how to instrument in your language, read one of our language-specific guides below or check out our tracing examples.

Planning instrumentation for distributed tracing 🔗

For customers who have not yet instrumented their applications, our default recommendation is to use OpenTracing to instrument and to use Jaeger tracer libraries to implement the OpenTracing APIs and report spans to SignalFx (via the Smart Agent and Smart Gateway). OpenTracing has a rich ecosystem of instrumentations for most often used libraries and frameworks in a growing spectrum of programming languages. Both OpenTracing and Jaeger are active CNCF projects with a rapidly growing community behind them.

RPC layer(s) and service/web framework(s) are the best places to start when thinking about how to go about instrumenting your application. Both of these will likely have a large coverage area and touch a significant number of transaction paths to give you baseline tracing coverage and visualize end-to-end traces and dependencies between your services.

Next, you should identify services critical to your business and look for areas not covered by RPC or service/web frameworks. Within these services, identify high value transactions and critical paths and start by instrumenting these.

The information listed below can provide more suggestions and food for thought.

Instrumentation guidelines and limitations 🔗

Span names 🔗

  • Span name must be present and be a valid dimension value
  • Span local endpoint service name must be present and be a valid dimension value
  • Span names must be low-cardinality; variable components should be recorded as tags