Use the Splunk Universal Forwarder with the Collector 🔗
Splunk Enterprise Cloud and Splunk Observability Cloud currently use different data collection agents:
Enterprise Cloud uses the Splunk Universal Forwarder (UF) to capture logs and some metrics (stored as logs).
Observability Cloud uses OpenTelemetry to capture traces, metrics, and logs. Logs are currently captured through bundled Fluentd.
You can manage your data ingestion manually by deploying the Splunk Distribution of OpenTelemetry Collector alongside the UF on each virtual machine (VM).This solution is applicable for VM environments for operating systems that are currently supported by both Observability Cloud and Enterprise and Cloud, running in common environments such as AWS EC2, GCE, Azure VMs, and VMWare.
For Kubernetes deployments, use the Splunk Distribution of OpenTelemetry Collector for Kubernetes. Install the Collector using the method that best suits your needs:
The benefits of using this solution are:
You can use Observability Cloud alongside Enterprise or Enterprise Cloud without capturing and submitting any duplicate telemetry data.
When used with Splunk Log Observer Connect, you can take advantage of effectively all Observability Cloud logging functionality, including Related Content.
You do not have to update existing UF deployments.
Collect data with the Collector and Universal Forwarder 🔗
To collect data with the Collector and the UF:
Configure each agent using the default configuration files:
Run the following command to skip installation of Fluentd and the plugins and dependencies for the Collector:
curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \ sudo sh /tmp/splunk-otel-collector.sh --realm SPLUNK_REALM -- SPLUNK_ACCESS_TOKEN --without-fluentd
Ensure that the UF captures the fully qualified domain name (FQDN) of the host, which is used to identify hosts in Observability Cloud. The UF can already capture this, and its behavior is consistent with the Collector. To capture the FQDN:
$SPLUNK_HOME/etc/system/local/directory, open server.conf and verify that the following stanza is present:
[general] hostnameOption = fullyqualifiedname
$SPLUNK_HOME/etc/system/local/ directorydirectory, open inputs.conf and verify that the following stanza is present:
Restart the UF.
Ensure that the UF captures the name of the service, which you must set manually in the Collector configuration and within your applications.
For the UF, do this in the same way that you append trace and span IDs to logs.
To capture the name of the service, set the
OTEL_SERVICE_NAMEenvironment variable in the configuration file. On Linux, run
export OTEL_SERVICE_NAME=<yourServiceName>. On Windows Powershell, run
$env:OTEL_SERVICE_NAME=<yourServiceName>. See https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md#environment-variables on GitHub to view additional OpenTelemetry specification environment variables.
Restart both agents.