Link logs and other resources to SignalFx µAPM service, traces, and spans 🔗
Important
The original µAPM product, released in 2019, is now called µAPM Previous Generation (µAPM PG). In the documentation, µAPM now refers to the product released on March 31, 2020.
If you are using µAPM Previous Generation (µAPM PG), see µAPM PG Traces, Spans, Metrics, and Metadata.
Create global data links for services, trace IDs, or span IDs to associate other resources with µAPM properties.
µAPM supports linking to other resources with only global data links. You can’t link other resources to µAPM properties with local data links.
With data links, you can link µAPM services, traces, or spans to custom SignalFx dashboards, Splunk instances, or a custom URL of your choosing.
To view data links you created for µAPM, see View linked resources for SignalFx µAPM properties.
Data links for services are slightly different than for inferred services. You have to create a data link that links to a SignalFx dashboard for a specific inferred service for the service to have a top-ranked SignalFx dashboard. Otherwise, the process is the same as for standard services. For more information, see Connect databases and inferred services to SignalFx dashboards.
Prerequisites 🔗
You have to be an administrator in your SignalFx organization to create data links and associate resources with µAPM properties.
Parameters to create data links for µAPM properties 🔗
When you create a data link, you have to specify fields that represent µAPM properties for the data link Trigger parameter. When you create data links for µAPM properties, SignalFx can transfer available context parameter values for the property to SignalFx dashboards and custom URLs. These are the fields for µAPM properties and their context parameters:
| Property | Field | Context parameters | Description |
|---|---|---|---|
| Service | sf_service |
sf_environment, sf_operation, sf_endpoint |
The name of the service in SignalFx. This is generally specified when configuring instrumentation for the service. |
| Trace ID | trace_id |
sf_service, sf_operation, any global tags for the trace |
A unique 16 or 32-character identifier associated with a specific trace. |
| Span ID | span_id |
sf_service, sf_operation, trace_id, span_id, all tags for the span |
A unique 16-character identifier associated with a specific span. |
| Span tag | An existing span tag value. | sf_service, sf_operation, span_id, trace_id, all other tags for the span |
A field-value pair that describes the span. Span tags are generally specified when configuring instrumentation for the corresponding service, and can also be specified in the Smart Agent or OpenTelemetry Collector configuration. |
Connect µAPM to Splunk logs 🔗
Create a data link that connects information about µAPM services, traces, and spans to Splunk searches. This means you can create a resource that runs a Splunk search to parse logs for any service, trace ID, or span ID you’re analyzing. Specify either a Splunk Cloud or Splunk Enterprise instance.
You have to create a data link for each µAPM property to connect logs in Splunk. For example, if you want to connect Splunk logs to trace IDs and span IDs, you have to create two data links, one that carries the context for trace IDs to Splunk and one that carries the context for span ID’s to Splunk.
Follow these steps to create a data link for trace IDs that runs a Splunk search query for log events that include a specific trace ID in a Splunk instance. The process is the same for services, span IDs, and span tags - just substitute the Trigger for the property you want to create a data link for.
- In SignalFx, go to Settings > Organization Settings > Global Data Links.
- Click New Link.
- Enter a Link Label. This is what you select when you want to use the data link to drill down into a specific trace ID. For example, you could enter
Splunk Cloud Search. - For Link to, select
Splunk. - For the Trigger, select
Any Value ofand entertrace_id. - Enter the Splunk instance fully qualified domain name (FQDN) and port of your Splunk instance for the URL. For example, you could enter
https://<yourHostname>.splunkcloud.com:443for a Splunk Cloud instance. You could also specify the FQDN and port for a Splunk Enterprise instance you have access to. - Keep the Minimum Time Window at
1m. - If your Splunk instance refers to fields differently than SignalFx refers to them, associate SignalFx fields with related fields in Splunk. For example,
sf_servicein SignalFx could beservicein Splunk. If something like this is the case, specify the SignalFx Term and External Term. Otherwise, skip this step. - Click Save. When you view a specific trace, you can drill down into this data link and view a Splunk search that includes all log events with the trace ID within the time range of the trace.
Connect µAPM to Kibana logs 🔗
Create a data link that connects information about µAPM services, traces, and spans to Kibana. Pass µAPM properties and their characteristics in a Kibana URL to transfer context from SignalFx to Kibana.
Follow these steps to create a data link for a log filter in Kibana for a selected trace ID. You can also filter on other µAPM properties - just subsitute the Trigger for the property you want to create a log filter in Kibana for.
- In SignalFx, go to Settings > Organization Settings > Global Data Links.
- Click New Link.
- Enter a Link Label. This is what you select when you want to use the data link to drill down into a specific trace ID. For example, you could enter
Kibana filter. - For Link to, select
Kibana. - For the Trigger, select
Any Value ofand entertrace_id. - Enter a Kibana URL that includes the
trace_idfield in a log filter for the URL. For example, you could enter this URL:
http://<yourKibanaFQDN>/kibana/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'{{start_time}}',mode:absolute,to:'{{end_time}}'))&_a=(columns:!(_source),interval:auto,query:(language:kuery,query:'traceId:{{value}}'),sort:!('@timestamp',desc))
- Enter your preferred Time Format.
- Keep the Minimum Time Window at
1m. - If Kibana refers to fields differently than SignalFx refers to them, associate SignalFx fields with related fields in Kibana. For example,
sf_servicein SignalFx could beservicein Kibana. If something like this is the case, specify the SignalFx Term and External Term. Otherwise, skip this step. - Click Save. When you view a specific trace, you can drill down into this data link and view a Splunk search that includes all log events with the trace ID within the time range of the trace.
Transfer µAPM context in a custom URL 🔗
You can transfer context of µAPM services, traces, spans, and tags you’re viewing in a custom URL. For parameters that you can use to transfer context in custom URLs, see Parameters to create data links for µAPM properties.
For example, you could specify a custom URL like this to transfer the context of a service you’re viewing to a URL of your choosing:
https://anexternalsite.com/search/?field={{key}}&value={{value}}&service={{sf_service}}&st={{start_time}}&et={{end_time}}
For more information about creating data links with a custom URL, see Configuring global data links.
Connect µAPM to a SignalFx dashboard 🔗
Connect a service, trace, span, or tag with custom dashboards available in SignalFx. When you drill down into a data link that sends you to a SignalFx dashboard, the entire context of the property you were viewing transfers to the SignalFx dashboad. For example, if you’re viewing a service, a data link transfers information about any endpoints you filtered for, including any filters for selected endpoints, the selected environment, and any tags you filtered for within the selected time range to the SignalFx dashboard.
For information about creating dashboards in SignalFx, see Creating, Sharing, and Protecting Dashboards.
Follow these steps to associate a data link for a service to a SignalFx dashboard. You can set up a data link for any service, or a specific service.
- In SignalFx, go to Settings > Organization Settings > Global Data Links.
- Click New Link.
- Enter a Link Label. This is what you select when you want to use the data link to drill down into a specific service. For example, you could enter
My better dashboard. - For the Trigger, select
Any Value ofand entersf_serviceto associate the data link with every service. If you want to create the data link for a specific service, selectProperty:Value Pairinstead and entersf_service:<yourServiceName>for the service you want to create the data link for. - Click Choose Dashboard and select the SignalFx dashboard you want to associate with the data link.
- Click Save. When you view a service that matches the Trigger, you can carry the entire context of the service to the SignalFx dashboard.
Connect databases and inferred services to SignalFx dashboards 🔗
Create a data link specifically for a single inferred service to associate a SignalFx dashboard with the inferred service as the top-ranked dashboard. The top-ranked SignalFx dashboard is the View Dashboard option in the Monitoring tab when you view a service from the service list or service map. Triggers for data links for SignalFx dashboards that use wildcards (*) for service names can’t be top-ranked dashboards for inferred services.
For example, a SignalFx dashboard associated with a data link that contains a trigger sf_service:* can’t be a top-ranked SignalFx dashboard for an inferred service. To create a data link that acts as a default SignalFx dashboard for an inferred service from the Monitoring tab, the trigger has to include the name of the inferred service. To create a data link for a default SignalFx dashboard for an inferred service mydb, the trigger has to be sf_service:mydb.