Docs » Monitor services and hosts in Splunk Infrastructure Monitoring » Monitor Amazon Web Services » Monitor Amazon Web Services

Monitor Amazon Web Services 🔗

The Infrastructure Monitoring Amazon Web Services (AWS) integration imports metrics and metadata from AWS CloudWatch and the following AWS services, as well as other applications.

Metrics are data points identified by a name; and metadata is information that helps you identify aspects of the metrics such as its source. AWS metrics and metadata help you monitor and troubleshoot the AWS services you’re using, such as AWS EC2. The metrics and metadata also help you monitor applications, such as Kubernetes clusters, that use the AWS services.

About AWS data 🔗

Refer to the AWS official documentation for a list of the available AWS metrics and other data, or read about the metadatada we provide for AWS.

By default, Observability Cloud will bring in data from all supported AWS services associated with your account, with certain limitations. To manage the amount of data to import, see Specifying data and metadata to import.

AWS namespaces 🔗

Infrastructure Monitoring imports AWS namespace metadata in the using the dimension namespace. For most AWS services, the namespace name has the form "AWS/<NAME_OF_SERVICE>", such as “AWS/EC2” or “AWS/ELB”. To select a metric time series (MTS) for an AWS metric when the metric has the same name for more than one service, such as CPUUtilization, use the namespace dimension as a filter.

To control the amount of data you import, specify the namespaces you want to import as well as the data you want to import or exclude from each namespace. For more information, see Specifying data and metadata to import.

Uniquely identifying AWS instances 🔗

The AWS instance ID is not a unique identifier. To uniquely identify an AWS instance, you need to concatenate the instanceId, region, and accountID dimension values, separated by underscores “_”, as shown in the following example:

instanceId_region_accountID

To construct the identifier manually, first get the specified values for each of your instances. For example, you can use the following cURL command:

curl http://<INSTANCE_URL>/latest/dynamic/instance-identity/document

Here’s an example JSON response from the cURL command:

{
   "devpayProductCodes" : null,
   "privateIp" : "10.1.15.204",
   "availabilityZone" : "us-east-1a",
   "version" : "2010-08-31",
   "accountId" : "134183635603",
   "instanceId" : "i-a99f9802",
   "billingProducts" : null,
   "instanceType" : "c3.2xlarge",
   "pendingTime" : "2015-09-02T16:45:40Z",
   "imageId" : "ami-2ef44746",
   "kernelId" : null,
   "ramdiskId" : null,
   "architecture" : "x86_64",
   "region" : "us-east-1"
}

From the response, copy the values for instanceId, region, and accountId, then concatenate them with underscores as separators.

Use the resulting string identifier as the value for the sfxdim\_AWSUniqueId dimension.

Import AWS CloudWatch data and metadata 🔗

Infrastructure Monitoring queries AWS CloudWatch to import (or download) metrics, logs, and metadata. During this import, Infrastructure Monitoring gives the metrics special names so you can identify them as coming from AWS. In Infrastructure Monitoring, AWS metadata becomes dimensions and custom properties. AWS tags are key-value pairs, so Infrastructure Monitoring converts them to custom properties.

Importing data and metadata from applications 🔗

Infrastructure Monitoring also imports metrics, metadata, and logs for some of your applications that use AWS services. The following table lists these applications.

Get data in

Monitor

Description

Collect Kubernetes data

Monitor Kubernetes

Import metrics and logs from Kubernetes clusters running in EC2 instances or EKS.

Monitor hosts

Import metrics and logs from Linux and Windows hosts running in EC2 instances.

Instrument back-end applications to send spans to Splunk APM

Introduction to Splunk APM

Import application metrics and spans running in hosts, Kubernetes clusters, or Lambda functions.

Specifying data and metadata to import 🔗

The AWS integration imports metrics from a list of supported AWS services in all built-in AWS namespaces. To limit the amount of AWS data that the integration imports, specify a subset of built-in namespaces from which you need data. For each namespace, you can then filter the data based on AWS tags or metric names or both.

Refer to the section Amazon Web Services to see the list of AWS services from which the AWS integration imports data.

You can also limit the amount of AWS data that the integration imports by changing the rate at which Infrastructure Monitoring polls AWS CloudWatch.

Note

You must be an administrator of your AWS account to choose namespaces and set filters.

  • To select the built-in namespaces for which you want data, click Select namespaces, then choose the namespaces.

  • Infrastructure Monitoring also lets you import data from custom namespaces. To specify a custom namespace from which you want data, click Add custom namespaces, type the name of the custom namespace, then press Enter. Using this procedure, you can specify multiple custom namespaces.

Specifying filters for AWS data you want to import doesn’t affect tag syncing.

Example: Specify namespaces and filters 🔗

The following example demonstrates how to specify the following:

  • Namespace: Only import data from Amazon ElasticSearch Service and EC2

  • Data filters: Only import data from EC2 if it matches a filter

  • Tag filters: Exclude data from resources that have the AWS tag version:canary

To create these specifications, perform the following steps:

  1. From the list of namespaces, select Amazon ElasticSearch Service and EC2.

  2. To limit the data Infrastructure Monitoring imports from EC2, click the drop-down arrow to see the data filters.

  3. To select the filters you want from the following options:

    • Use Import only if you want to specify a filter for the data to import.

    • Use Don’t import if you want to specify a filter for the data to exclude.

  4. To use AWS tags to limit the data Infrastructure Monitoring imports, filter by tag. For this example, specify a filter that excludes data from resources that have the AWS tag version:canary.

Infrastructure Monitoring adds the prefix aws_tag_ to the names of tags imported from AWS, which indicates their origin. For example, the AWS tag version:canary appears in Infrastructure Monitoring as aws_tag_version:canary. When you filter an AWS integration by tag, enter the name of the tag as it appears in AWS.

You can also choose specific metrics to include or exclude. For example, consider the following conditions.

../../_images/aws-metric-tag.png

Only metricA and metricB are included, and only for resources specified by the tags:

  • For a resource that has the tag env:prod or env:beta, metricA and metricB are included.

  • For a resource that doesn’t have the tags env:prod or env:beta, no metrics are included.

  • No other metrics are included.

Infrastructure Monitoring supports wildcards in filters. For example, if you want to import data for a resource that has specific tags, regardless of the tag values, specify this filter:

../../_images/aws-metric-tag-wildcard.png

In this example, metricA and metricB are included for resources that have the env tag set to any value. No other metrics are included.

You can use the Actions menu next to a namespace name to copy or paste filters from one namespace to another, clear the filters for the namespace, or remove the namespace from the list of namespaces to include. When you remove a namespace, Infrastructure Monitoring no longer includes metrics from that namespace.

When you finish specifying the namespaces, metrics, and tags to include or exclude, click Save.

Note

You can specify more complex filtering options for a namespace by using the Infrastructure Monitoring API. In this case, the UI displays a message indicating that the filter is defined programmatically. To see which metrics and tags are included or excluded for that namespace, click View filter code.

Import specific AWS CloudWatch metric sources 🔗

To import some AWS CloudWatch metrics, you need to configure AWS CloudWatch as well as Infrastructure Monitoring.

Receiving S3 metrics 🔗

For S3, Infrastructure Monitoring defaults to receiving the daily storage metrics listed on the Amazon S3 console page. Amazon bills you separately for the request metrics shown on that page, so you must explicitly select to import them. To learn more about selecting them, see the AWS S3 documentation.

Infrastructure Monitoring also imports metadata for AWS S3. To learn more, see Amazon Simple Storage Service (S3) metadata.

Receiving metrics via the Cloudwatch agent 🔗

AWS provides a CloudWatch agent that lets you import more system-level metrics from Amazon EC2 instances and also lets you collect system-level metrics from on-premises servers. To import these metrics in Infrastructure Monitoring, add the namespace you use for the AWS CloudWatch agent as a custom namespace in your AWS integration, as described in the section Specifying data and metadata to import).

To learn more about the AWS CloudWatch agent, see the AWS documentation.

Monitor AWS services and identify problems 🔗

Visit the Infrastructure page to monitor the health of the AWS services you’re using. It provides a key metric for each service. You can also drill down into specific instances of an AWS service. For example, start by viewing the key metrics for your EC2 service, and then filter for a specific instance ID to analyze the EC2 instance with that ID.

Follow these steps to find and troubleshoot AWS services from the Infrastructure page:

  1. Select Navigation menu > Infrastructure, then click Amazon AWS category.

  2. Select the specific service you want to analyze. For example, click EBS to view information about your storage volumes. If you see the message No Data Found, you first need to configure the integration for the service.

  3. Compare instances of the services to investigate their relative health. Select a metric from the Color by drop-down list. In the heat map, colors indicate the health of each instance based on the selected metric. For example, consider an AWS EBS heat map for the total number of I/O operations in a time period (Total IOPS). The heat map displays high Total IOPS in lighter colors, which indicates that the instances are healthy. In comparison, the heat map displays low IOPS in a darker color, which indicates that the instances have a I/O-related problem.

    If the heat map only uses green and red, then green indicates a healthy instance and red indicates a problem.

    To apply visually-accessible color palettes to heat maps, select <USER-ID> > Account Settings, then select your desired color accessibility from the Color Accessibility menu.

  4. Investigate correlations between instances and their health by grouping the instances based on a dimension, custom property, or tag. To group instances, select the metadata name from the Group by drop-down list.

    Note

    In the DynamoDB navigator, when you view the heatmap and group the instances by aws_account_id, some entries might report back as “n/a” because properties are omitted when the query is not specific enough. To work around this issue, filter by Operation, then group by aws_account_id.

  5. Outliers are another indication of instance health. An outlier is a metric value that is significantly outside the mean or median value of all other metric values. To find the outliers in metrics coming from AWS services, use the Find Outliers setting and specify the Scope and Strategy:

    You can select one of two Strategies to find outliers, as described in the following table.

    Strategy

    Description

    Deviation from Mean

    Instances shown in red are ones that exceed the mean value of the metric by at least three standard deviations.

    Deviation from Median

    Instances shown in red are ones that exceed the median absolute deviation value by at least three absolute deviations. Deviation from Median This setting does not weigh extreme outliers as heavily as the standard deviation.

  6. To drill down to a specific instance you want to investigate, hover over the heatmap to find the specific instance ID, then click the cell to see the information for that ID. For every instance, Infrastructure Monitoring provides a default dashboard.

The default dashboard helps you analyze all the available metadata about the cloud service the instance is running in, the instance itself, and any custom tags associated with the instance. The default dashboard provides metric time series (MTS) for key metrics.

Use default dashboards to monitor AWS services 🔗

Observability Cloud provides default dashboards for supported AWS services. Default dashboards are available in dashboard groups based on the AWS service a dashboard represents data for.

To find default dashboards for AWS services, select Navigation menu > Dashboards and search for the AWS service you want to view dashboards for.

Explore built-in content 🔗

To see all of the navigators provided for data collected in your organization, go to the Infrastructure page. To see all the pre-built dashboards for data collected in your organization, select Dashboards > Built-in.

Amazon EC2 instances are powered by their respective public cloud service as well as the Splunk Distribution of OpenTelemetry Collector. You need both for all the charts to display data in the built-in dashboards.

  • If you have only the public cloud service and the Smart Agent configured, some charts in the built-in dashboards for Amazon EC2 instances display no data.

  • If you have only the public cloud service configured, you can see all the cards representing the services where data come from, but some charts in the built-in dashboards for Amazon EC2 instances display no data.

  • If you have only Smart Agent configured, Amazon EC2 instance navigator isn’t available.

Filter AWS data using tags 🔗

You can filter AWS data using AWS tags, but only with namespaces for which Infrastructure Monitoring syncs tags. For more information, see AWS namespaces. For example, if you use Detailed Monitoring for EC2 instances in AWS, Infrastructure Monitoring imports the following dimensions:

  • AutoScalingGroupName

  • ImageId

  • InstanceId

  • InstanceType.

Note

Unsupported characters within a dimension key are converted to underscores.

You can use the following AWS metadata to filter metrics:

Custom Property

Form

Description

aws_account_id

key-value pair

AWS account ID for the instance, volume or load balancer. Use this property to differentiate between metrics you import.

aws_tag_<TAGNAME>

key and optional value

AWS custom tag name for the instance, volume or load balancer. A metric may have more than one associated custom tag name.

Use aws_account_id to differentiate between metrics you import from multiple AWS accounts. Infrastructure Monitoring adds aws_account_id as a dimension of the MTS for the metric.

For supported AWS services, Infrastructure Monitoring imports AWS tags and adds them as custom properties to the MTS for the metric. For example, if AWS tag has the value named Production, it will be shown in Infrastructure Monitoring as aws_tag_Production.

CloudWatch rollups and Infrastructure Monitoring MTS 🔗

AWS CloudWatch uses rollups to summarize metrics, and it refers to them as “statistics”. To learn more about rollups, see Rollups in data resolution and rollups in charts.

Because AWS CloudWatch rollups don’t map directly to Infrastructure Monitoring rollups, you can’t directly access AWS CloudWatch rollups using the rollup selection menu in the Chart Builder. Instead, Infrastructure Monitoring captures the rollups as individual MTS that have the dimension stat.

AWS statistic

IM dimension

Definition

Average

stat:mean

Mean value of metric over the sampling period

Maximum

stat:upper

Maximum value of metric over the sampling period

Minimum

stat:lower

Minimum value of metric over the sampling period

Data Samples

stat:count

Number of samples over the sampling period

Sum

stat:sum

Sum of all values that occurred over the sampling period

To use an AWS CloudWatch metric in a plot, always specify the following:

  • AWS Cloudwatch metric name

  • Filter for the stat dimension value that’s appropriate for the metric you’ve chosen.

For example, if you are using the metric NetworkPacketsIn for EC2 metrics, the only meaningful AWS statistics are Minimum, Maximum and Average. To plot NetworkPacketsIn metric with the rollup you want, filter for the stat dimension with a value that corresponds to the AWS statistic (rollup) value:

  • lower: Rollup that corresponds to the AWS rollup Minimum

  • upper: Rollup that corresponds to the AWS rollup Maximum

  • mean: Rollup that corresponds to the AWS rollup Average

Note

The “Rollup: Multiple” label in a plot for a CloudWatch metric indicates that you haven’t specified the rollup you want. To avoid confusion, specify the rollup as soon as possible.

Infrastructure Monitoring uses a sixty-second sampling period for metrics it imports from AWS.

To learn more, see the AWS developer documentation for AWS CloudWatch.