Docs » Transform your data with log processing rules

Transform your data with log processing rules 🔗

Note

Only customers with a Splunk Log Observer entitlement in Splunk Observability Cloud can create or manage log processing rules. If you do not have a Log Observer entitlement and are using Splunk Log Observer Connect instead, see Introduction to Splunk Log Observer Connect to learn what you can do with the Splunk Enterprise integration.

Add value to your raw logs by creating log processing rules, also known as processors, to transform your data or a subset of your data as it arrives.

To add more control to processors, you can add filters that determine which logs a processor will be applied to.

On the Logs Pipeline Management page, you can adjust the order in which your processing rules run, edit processors, or delete processors.

Note

You can’t edit or delete prepackaged processors.

Prepackaged processors appear at the beginning of the list of processors, and they’re identified by a lock icon. These prepackaged processors always execute before any processors you define. You can’t modify or reorder prepackaged processors.

One example of a prepackaged processor is the Level to severity attributed remapper.

Splunk Observability Cloud includes prepackaged processors for Kubernetes and Cassandra.

Observability Cloud provides three types of log processors:

Order of execution of logs pipeline rules 🔗

Logs pipeline rules execute in the following order:

  1. All log processing rules (field extraction, field copy, and field redaction processors)

  2. All log metricization rules

  3. All Infinite Logging rules

Because log processing rules execute first, you can create field extraction rules, then use the resulting fields in log metricization rules or Infinite Logging rules or both. For more information, see Sequence of logs pipeline rules.

Field Extraction Processors 🔗

Field extraction lets you find an existing field in your incoming logs and create a processor based on the format of the field’s value.

Field extraction helps you do the following tasks:

Consider the following raw log record

10.4.93.105 - - [04/Feb/2021:16:57:05 +0000] “GET /metrics HTTP/1.1” 200 73810 “-” “Go-http-client/1.1” 23

If you have not defined any processors in your logs pipeline, you can only do a keyword search on the sample log, which searches the _raw field. The following table shows how you can extract fields to define processing rules:

Example of value to extract

Processor definition to use

IP address (10.4.93.105)

IP

04/Feb/2021:16:57:05 +0000

time

GET

method

/metrics

path

Creating Regex and Event Time field extractions allows you to filter and aggregate on the fields: IP, time, method, and path. This enables you to create the query “Display a Visual Analysis of the number of requests from {IP} broken down by {method}”.

Additionally, the extracted fields begin appearing in the Fields summary panel along with their top values and other statistics.

There are three types of field extraction. These are:

  • Regex Processors

  • JSON Processors

  • Event Time Processors

  • KV Parser Processors

To start creating a field extraction, follow these steps:

  1. From the navigation menu, go to Organization Settings > Logs Pipeline Management. A list of existing processors is displayed with the prepackaged processors displaying first.

  2. Click New Processing Rule.

    Alternatively, you can launch the processor wizard from Log Observer. To do this, click into a log in the Logs table. The Log Details panel appears on the right. Click a field value then select Extract field. This takes you to Define Processor, the second step of the processor wizard. Skip to step 7.

  3. Select Field Extraction as the processor type, then click Continue. This takes you to Select sample, the first step in the processor wizard.

  4. To narrow your search for a log that contains the field you want to extract, you can select a time from the time picker or click Add Filter and add keywords or fields.

  5. Click the log containing the field you want. A list of fields and values appears below the log line.

  6. Click Use as sample next to the field you want to extract, then click Next. This takes you to Define Processor, the second step of the processor wizard.

  7. Select the extraction processor type that you want to use.

  8. From here, follow the steps to create the extraction processor type you selected:

Create a Regex Processor 🔗

The regular expression workspace lets you to extract fields from your data and then create a new processor using regex. Pipeline Management makes suggestions to help you write the appropriate regex for your processor. You can modify the regex within the processor wizard.

To create a Regex Processor, follow these steps:

  1. Highlight the value of the field you want to extract in your sample and select Extract field from the drop-down menu.

  2. Click into the field name box and enter a name for the field you selected. The default name is Field1. Results display in a table.

  3. Click Edit regex below the field name box if you want to modify the regex that the processor has automatically generated to create this rule based on your field name and value.

  4. Preview your rule in the table to ensure that the correct fields are extracted.

  5. To apply your new rule to only a subset of incoming logs, add filters to the content control bar. The new rule will apply only to logs matching this filter.

  6. In step 3 of the processor wizard entitled Name, Save, and Review, give your new rule a name and description.

  7. Review your configuration choices, then click Save. Your processor defaults to Active and immediately begins processing incoming logs.

  8. To see your new processor, go to Organization Settings > Logs Pipeline Management, expand the Processing Rules section, and find it in the list. You can reorder, edit, or delete all processors except those that are prepackaged (shown with a lock). To disable your processor, click Inactive.

Create a JSON Processor 🔗

To create a JSON Processor, follow these steps:

  1. To apply your new rule to only a subset of incoming logs, click Add Filter and add a keyword or field. The new rule will apply only to logs matching this filter. Pipeline Management only applies the new processor to log events that match this filter.

  2. Preview your rule to ensure that Pipeline Management is extracting the correct field values.

  3. If you see the correct field values in the results table, click Next. Otherwise, adjust your filter.

  4. Add a name and description for your new rule, then click Save. Your processor defaults to Active and immediately begins processing incoming logs.

  5. To see your new processor, go to Organization Settings > Logs Pipeline Management, expand the Processing Rules section, and find it in the list. You can reorder, edit, or delete all processors except those that are prepackaged (shown with a lock). To disable your processor, click Inactive.

Create an Event Time Processor 🔗

To create an Event Time Processor, follow these steps:

  1. Select a time format from the drop-down list. The wizard looks for the selected format within your sample.

  2. From the matches you see, select the time when the sample event occurred, then click Next.

  3. Add filters to the content control bar to define a matching condition, then click Next. Pipeline Management only applies the new processor to log events that match this filter.

  4. Give your new rule a name and description.

  5. Review your configuration choices, then click Save. Your processor defaults to Active and immediately begins processing incoming logs.

  6. To see your new processor, go to Organization Settings > Logs Pipeline Management, expand the Processing Rules section, and find it in the list. You can reorder, edit, or delete all processors except those that are prepackaged (shown with a lock). To disable your processor, click Inactive.

Create a KV Parser Processor 🔗

A KV Parser Processor is a rule that parses key-value (KV) pairs. To create a KV Parser Processor, follow these steps:

  1. To apply your new rule to only a subset of incoming logs, click Add Filter then add a keyword or field. The new rule will apply only to logs matching this filter.

  2. Preview your rule to ensure that Pipeline Management is extracting the correct field values.

  3. If you see the correct field values in the results table, click Next. Otherwise, adjust your filter.

  4. Add a name and description for your new rule, then click Save. Your processor defaults to Active and immediately begins processing incoming logs.

  5. To see your new processor, go to Organization Settings > Logs Pipeline Management, expand the Processing Rules section, and find it in the list. You can reorder, edit, or delete all processors except those that are prepackaged (shown with a lock). To disable your processor, click Inactive.

Field Copy Processors 🔗

Field Copy Processors let you define a new relationship between new or existing fields. One way to use Field Copy Processors is to use OpenTelemetry mappings to help power your Related Content suggestions.

To create a Field Copy Processor, follow these steps:

  1. From the navigation menu, go to Organization Settings > Logs Pipeline Management.

  2. Click New Processing Rule.

  3. Select Field Copy, then click Continue.

  4. Enter a target field in the first text box. You can choose from available extracted fields in the drop-down list.

  5. In the second text box, choose a field to which you want to map your target field. The drop-down list options suggest OpenTelemetry mappings, which help power your Related Content suggestions.

  6. If you want to create multiple mappings, click + Add another field copying rule and repeat steps 4 and 5; otherwise, click Next.

  7. To apply your new rule to only a subset of incoming logs, add filters to the content control bar. The new rule is applied only to logs matching this filter. If you do not add a filter, the rule is applied to all incoming log events.

  8. Preview your rule to ensure that Pipeline Management is extracting the correct field values, then click Next.

  9. Give your new rule a name and description, then click Save. Your processor defaults to Active and immediately begins processing incoming logs.

  10. To see your new processor, go to Organization Settings > Logs Pipeline Management, expand the Processing Rules section, and find it in the list. You can reorder, edit, or delete all processors except those that are prepackaged (shown with a lock). To disable your processor, click Inactive.

Field Redaction Processors 🔗

Field redaction lets you mask data, including personally identifiable information.

To create a Field Redaction Processor, follow these steps:

  1. From the navigation menu, go to Organization Settings > Logs Pipeline Management.

  2. Click New Processing Rule.

  3. Select Field Redaction, then click Continue. This takes you to the first step in the processor wizard, Select Sample.

  4. To find a log that contains the field you want to redact, add filters to the content control bar until the Logs table displays a log with the desired field.

  5. Click the log containing the field you want. A list of fields and values appears below the log line.

  6. Click Use as sample next to the field you want to redact, then click Next. This takes you to Define Processor, the second step of the processor wizard.

  7. Select if you want to redact an entire field value or a partial field value. If you want to redact a partial field value, highlight the portion you want to redact. You can edit the regex here.

  8. Define a matching condition. To apply your new rule to only a subset of incoming logs, add filters to the content control bar. The new rule will apply only to logs matching this filter.

  9. Give your new rule a name and description.

  10. Review your configuration choices, then click Save. Your processor defaults to Active and immediately begins processing incoming logs.

  11. To see your new processor, go to Organization Settings > Logs Pipeline Management, expand the Processing Rules section, and find it in the list. You can reorder, edit, or delete all processors except those that are prepackaged (shown with a lock). To disable your processor, click Inactive.

Note

If the field you redacted also appears in _raw, it is still available in _raw. Redact the field in _raw in addition to redacting the field itself.