Link APM services, traces, and spans to relevant resources 🔗
Create global data links to link Splunk APM properties, such as services, traces, spans, and span tags, to relevant resources. For example, you can link APM properties to Infrastructure Monitoring dashboards, Splunk instances, Kibana logs, or custom URLs.
Global data links are also useful because they can dynamically transfer contextual information about the property you’re viewing to the linked resource, helping you get to relevant information faster.
For information about how to access global data links, see Access relevant resources linked to APM services, traces, and spans.
Prerequisite 🔗
You must be an administrator to create global data links.
Parameters to create global data links for APM properties 🔗
When you create a global data link for an APM property, you can transfer the property’s context parameter values to dashboards and custom URLs. Here are the APM properties and their context parameters.
Property |
Field |
Context parameters |
Description |
|---|---|---|---|
Service |
|
|
The name of the service in APM. This is generally specified when configuring instrumentation for the service. |
Trace ID |
|
|
A unique 16- or 32-character identifier associated with a specific trace. |
Span ID |
|
|
A unique 16-character identifier associated with a specific span. |
Span tag |
An existing span tag value. |
|
A field-value pair that describes the span. Use span tags to describe the span. Specify the tags when you configure instrumentation for the service. You can also specify span tags in the Splunk Distribution of OpenTelemetry Collector (or the Smart Agent, now deprecated) configuration. |
Link APM properties to Splunk Infrastructure Monitoring dashboards 🔗
You can create a global data link that passes information about APM services, traces, and spans to custom dashboards available in Splunk Observability Cloud. When you click a global data link to an Infrastructure Monitoring dashboard, the entire context of the property you were viewing transfers to the dashboard. For example, if you were viewing a service, the global data link transfers information about any endpoints you filtered for, including any filters for selected endpoints, the selected environment or environments, and any tags you filtered for within the selected time range to the dashboard.
The following task describes how to create a global data link for a service that links to an Infrastructure Monitoring dashboard. You can set up a global data link for any service, or a specific service.
Open the Observability Cloud main menu.
Hover over Data Configuration and select Global Data Links.
Click New Link.
Enter a Link Label. This is what you select when you want to use the global data link to drill down into a specific service. For example, you could enter Trace Ingest Dashboard.
For Show On, select Any Value of and enter sf_service to associate the global data link with every service. If you want to create the global data link for a specific service, select Property:Value Pair instead and enter sf_service:<yourServiceName> for the service you want to create the global data link for.
If you want the global data link to display based on the Show On value and one or more additional conditions, click Add Conditions. To define a condition based on a property name, select Any Value of and enter a property name. To define a condition based on a property name and a specific value, select Property:Value Pair and enter a property name and value pair. If you define multiple conditions, all conditions must be met for the link to display.
Click Choose Dashboard and select the dashboard you want to associate with the global data link.
Click Save. When you view a service that matches the Show On value, you can carry the entire context of the service to the dashboard.
Link APM properties to Splunk platform logs 🔗
You can create a global data link that passes information about an APM service, trace, or span to a Splunk Cloud Platform or Splunk Enterprise search. This means you can create a global data link that runs a Splunk search query to parse logs for any service, trace ID, or span ID you’re analyzing.
Create a global data link for each APM property that you want to connect to logs in a Splunk instance. For example, if you want to connect a trace ID to a logs in a Splunk instance, create a global data link that carries the trace ID context to the Splunk instance. To connect a span ID to logs in a Splunk instance, create a global data link that carries the span ID context to the Splunk instance.
The following task describes how to create a global data link for a trace ID. The global data link runs a Splunk search query for log events that include a specific trace ID in a Splunk instance. The process is the same for creating a global data link for a service, span ID, or span tag: use a Show On value for the property you want to create a global data link for.
Open the Observability Cloud main menu.
Hover over Data Configuration and select Global Data Links.
Click New Link.
Enter a Link Label. This is what you select when you want to use the global data link to drill down into a specific trace ID. For example, you could enter Splunk Cloud Platform Search.
For Link to, select Splunk.
For Show On, select Any Value of and enter trace_id.
If you want the global data link to display based on the Show On value and one or more additional conditions, click Add Conditions. To define a condition based on a property name, select Any Value of and enter a property name. To define a condition based on a property name and a specific value, select Property:Value Pair and enter a property name and value pair. If you define multiple conditions, all conditions must be met for the link to display.
Enter the Splunk instance fully qualified domain name (FQDN) and port of your Splunk instance for the URL. For example, you could enter https://<yourHostname>.splunkcloud.com:443 for a Splunk Cloud Platform instance. You could also specify the FQDN and port for a Splunk Enterprise instance you have access to.
Keep the Minimum Time Window at 1m.
If your Splunk instance refers to fields differently than APM refers to them, associate APM fields with related fields in Splunk. For example,
sf_servicein APM could beservicein your Splunk instance. If something like this is the case, specify the Splunk Observability Cloud Term value and map it to an External Term value.Click Save. When you view a specific trace, you can drill down into this global data link and view a Splunk search that includes all log events with the trace ID within the time range of the trace.
Link APM properties to Kibana logs 🔗
You can create a global data link that passes information about an APM service, trace, or span to a Kibana URL. By passing APM properties and their characteristics in a Kibana URL, you can transfer context from Splunk Observability Cloud to Kibana.
The following task describes how to create a global data link for a log filter in Kibana for a selected trace ID. You can also filter on other APM properties: use a Show On value for the property you want to create a global data link for.
Open the Observability Cloud main menu.
Hover over Data Configuration and select Global Data Links.
Click New Link.
Enter a Link Label. This is what you select when you want to use the global data link to drill down into a specific trace ID. For example, you could enter Kibana filter.
For Link to, select Kibana.
For Show On, select Any Value of and enter trace_id.
If you want the global data link to display based on the Show On value and one or more additional conditions, click Add Conditions. To define a condition based on a property name, select Any Value of and enter a property name. To define a condition based on a property name and a specific value, select Property:Value Pair and enter a property name and value pair. If you define multiple conditions, all conditions must be met for the link to display.
Enter a Kibana URL that includes the trace_id field in a log filter for the URL. For example, you can enter a URL like this one:
http://<yourKibanaFQDN>/kibana/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'{{start_time}}',mode:absolute,to:'{{end_time}}'))&_a=(columns:!(_source),interval:auto,query:(language:kuery,query:'traceId:{{value}}'),sort:!('@timestamp',desc))Enter your preferred Time Format.
Keep the Minimum Time Window at 1m.
If Kibana refers to fields differently than APM refers to them, associate APM fields with related fields in Kibana. For example,
sf_servicein APM could beservicein Kibana. If something like this is the case, specify the Splunk Observability Cloud Term and External Term. If the field names are the same, skip this step.Click Save. When you view a specific trace, you can drill down into this global data link and view a Splunk search that includes all log events with the trace ID within the time range of the trace.
Transfer APM context in a custom URL 🔗
You can create a global data link that passes information about an APM service, trace, or span to a custom URL. For parameters that you can use to transfer context in custom URLs, see Parameters to create global data links for APM properties.
For example, you can specify a custom URL like this one to transfer the context of a service to a URL: https://www.example.com/search/?field={{key}}&value={{value}}&service={{properties.sf_service}}&st={{start_time}}&et={{end_time}}.
For more information about creating global data links to a custom URL, see Link metadata to related resources using global data links in Splunk Observability Cloud.
Link databases and inferred services to Infrastructure Monitoring dashboards 🔗
Create a global data link specifically for a single inferred service to associate a dashboard with the inferred service as the top-ranked dashboard. The top-ranked dashboard is the View Dashboard option in the Monitoring tab when you view a service from the service list or service map. Triggers for global data links for dashboards that use wildcards (*) for service names can’t be top-ranked dashboards for inferred services.
For example, a dashboard associated with a global data link that contains a Show On value of sf_service:* can’t be a top-ranked dashboard for an inferred service. To create a global data link that acts as a default dashboard for an inferred service from the Monitoring tab, the Show On value must include the name of the inferred service. For instance, if you are creating a global data link for a default dashboard for the inferred service mydb, the Show On value must be sf_service:mydb.