Docs » Connect Splunk APM to logs and other resources » Link logs and other resources to Splunk APM services, traces, and spans

---

The docs are moving!

The docs are moving to a new site! If you can't find the information you're looking for on this page, check the Splunk Observability Cloud Documentation.

Link logs and other resources to Splunk APM services, traces, and spans

Important

The original µAPM product, released in 2019, is now called µAPM Previous Generation (µAPM PG). Wherever you see the reference, µAPM now refers to the product released on March 31, 2020.

If you’re using µAPM Previous Generation (µAPM PG), see Overview of SignalFx Microservices APM Previous Generation (µAPM PG).

Create global data links for services, trace IDs, or span IDs to associate other resources with Splunk APM properties.

APM supports linking to other resources with only global data links. You can’t link other resources to APM properties with local data links.

With data links, you can link APM services, traces, or spans to custom dashboards, Splunk instances, or a custom URL of your choosing.

To view data links you created for APM, see View linked resources for Splunk APM services, traces, and spans.

Data links for services are slightly different than for inferred services. You have to create a data link that links to a dashboard for a specific inferred service for the service to have a top-ranked dashboard. Otherwise, the process is the same as for standard services. For more information, see Connect databases and inferred services to Infrastructure Monitoring dashboards.

Prerequisites

You have to be an administrator in your organization to create data links and associate resources with APM properties.

Parameters to create data links for APM properties

When you create a data link, you have to specify fields that represent APM properties for the data link Trigger parameter. When you create data links for APM properties, you can transfer available context parameter values for the property to dashboards and custom URLs. These are the fields for APM properties and their context parameters:

Property	Field	Context parameters	Description
Service	sf_service	sf_environment, sf_operation, sf_endpoint	The name of the service in APM. This is generally specified when configuring instrumentation for the service.
Trace ID	trace_id	sf_service, sf_operation, any global tags for the trace	A unique 16 or 32-character identifier associated with a specific trace.
Span ID	span_id	sf_service, sf_operation, trace_id, span_id, all tags for the span	A unique 16-character identifier associated with a specific span.
Span tag	An existing span tag value.	sf_service, sf_operation, span_id, trace_id, all other tags for the span	A field-value pair that describes the span. Span tags are generally specified when configuring instrumentation for the corresponding service, and can also be specified in the Smart Agent or OpenTelemetry Collector configuration.

Connect APM to Splunk platform logs

Create a data link that connects information about APM services, traces, and spans to Splunk platform searches. This means you can create a resource that runs a Splunk search to parse logs for any service, trace ID, or span ID you’re analyzing. Specify either a Splunk Cloud or Splunk Enterprise instance.

You have to create a data link for each APM property to connect logs in a Splunk instance. For example, if you want to connect Splunk instance logs to trace IDs and span IDs, you have to create two data links, one that carries the context for trace IDs to a Splunk instance and one that carries the context for span ID’s to a Splunk instance.

Follow these steps to create a data link for trace IDs that runs a Splunk search query for log events that include a specific trace ID in a Splunk instance. The process is the same for services, span IDs, and span tags - just substitute the Trigger for the property you want to create a data link for.

1. In the application, go to Settings > Organization Settings > Global Data Links.
2. Click New Link.
3. Enter a Link Label. This is what you select when you want to use the data link to drill down into a specific trace ID. For example, you could enter Splunk Cloud Search.
4. For Link to, select Splunk.
5. For the Trigger, select Any Value of and enter trace_id.
6. Enter the Splunk instance fully qualified domain name (FQDN) and port of your Splunk instance for the URL. For example, you could enter https://<yourHostname>.splunkcloud.com:443 for a Splunk Cloud instance. You could also specify the FQDN and port for a Splunk Enterprise instance you have access to.
7. Keep the Minimum Time Window at 1m.
8. If your Splunk instance refers to fields differently than APM refers to them, associate APM fields with related fields in Splunk. For example, sf_service in APM could be service in your Splunk instance. If something like this is the case, specify the SignalFx Term and External Term. Otherwise, skip this step.
9. Click Save. When you view a specific trace, you can drill down into this data link and view a Splunk search that includes all log events with the trace ID within the time range of the trace.

Connect APM to Kibana logs

Create a data link that connects information about APM services, traces, and spans to Kibana. Pass APM properties and their characteristics in a Kibana URL to transfer context from SignalFx to Kibana.

Follow these steps to create a data link for a log filter in Kibana for a selected trace ID. You can also filter on other APM properties - just substitute the Trigger for the property you want to create a log filter in Kibana for.

1. In the application, go to Settings > Organization Settings > Global Data Links.
2. Click New Link.
3. Enter a Link Label. This is what you select when you want to use the data link to drill down into a specific trace ID. For example, you could enter Kibana filter.
4. For Link to, select Kibana.
5. For the Trigger, select Any Value of and enter trace_id.
6. Enter a Kibana URL that includes the trace_id field in a log filter for the URL. For example, you could enter this URL:

http://<yourKibanaFQDN>/kibana/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'{{start_time}}',mode:absolute,to:'{{end_time}}'))&_a=(columns:!(_source),interval:auto,query:(language:kuery,query:'traceId:{{value}}'),sort:!('@timestamp',desc))

7. Enter your preferred Time Format.
8. Keep the Minimum Time Window at 1m.
9. If Kibana refers to fields differently than APM refers to them, associate APM fields with related fields in Kibana. For example, sf_service in APM could be service in Kibana. If something like this is the case, specify the SignalFx Term and External Term. Otherwise, skip this step.
10. Click Save. When you view a specific trace, you can drill down into this data link and view a Splunk search that includes all log events with the trace ID within the time range of the trace.

Transfer APM context in a custom URL

You can transfer context of APM services, traces, spans, and tags you’re viewing in a custom URL. For parameters that you can use to transfer context in custom URLs, see Parameters to create data links for APM properties.

For example, you could specify a custom URL like this to transfer the context of a service you’re viewing to a URL of your choosing:

https://anexternalsite.com/search/?field={{key}}&value={{value}}&service={{sf_service}}&st={{start_time}}&et={{end_time}}

For more information about creating data links with a custom URL, see Configuring global data links.

Connect APM to an Infrastructure Monitoring dashboard

Connect a service, trace, span, or tag with custom dashboards available in the application. When you drill down into a data link that sends you to an Infrastructure Monitoring dashboard, the entire context of the property you were viewing transfers to the dashboard. For example, if you’re viewing a service, a data link transfers information about any endpoints you filtered for, including any filters for selected endpoints, the selected environment, and any tags you filtered for within the selected time range to the dashboard.

For information about creating dashboards, see Creating, Sharing, and Protecting Dashboards.

Follow these steps to associate a data link for a service to a dashboard. You can set up a data link for any service, or a specific service.

1. In the application, go to Settings > Organization Settings > Global Data Links.
2. Click New Link.
3. Enter a Link Label. This is what you select when you want to use the data link to drill down into a specific service. For example, you could enter My better dashboard.
4. For the Trigger, select Any Value of and enter sf_service to associate the data link with every service. If you want to create the data link for a specific service, select Property:Value Pair instead and enter sf_service:<yourServiceName> for the service you want to create the data link for.
5. Click Choose Dashboard and select the dashboard you want to associate with the data link.
6. Click Save. When you view a service that matches the Trigger, you can carry the entire context of the service to the dashboard.

Connect databases and inferred services to Infrastructure Monitoring dashboards

Create a data link specifically for a single inferred service to associate a dashboard with the inferred service as the top-ranked dashboard. The top-ranked dashboard is the View Dashboard option in the Monitoring tab when you view a service from the service list or service map. Triggers for data links for dashboards that use wildcards (*) for service names can’t be top-ranked dashboards for inferred services.

For example, a dashboard associated with a data link that contains a trigger sf_service:* can’t be a top-ranked dashboard for an inferred service. To create a data link that acts as a default dashboard for an inferred service from the Monitoring tab, the trigger has to include the name of the inferred service. To create a data link for a default dashboard for an inferred service mydb, the trigger has to be sf_service:mydb.

---

Previous (Connect Splunk APM to logs and other resources)

Next (View linked resources for Splunk APM services, traces, and spans)