Docs » Integrations Guide » Monitor Amazon Web Services

Monitor Amazon Web Services

You can easily monitor Amazon Web Services (AWS) in SignalFx using AWS CloudWatch metrics. (For instructions on setting up this integration, see Connect to AWS CloudWatch.) SignalFx provides a robust integration with CloudWatch, has a CloudWatch-powered mode for the Infrastructure Navigator, and includes many built-in-dashboards to help you get started monitoring Amazon Web Services.

You can also monitor Amazon Web Services instances and the services running on them by using the SignalFx Smart Agent. The SignalFx collectd agent offers a much higher degree of customization than is possible with AWS CloudWatch, and may be preferable for instances where you want to see metrics at a finer resolution, or where detailed control over the metrics sent matters.

Note

The option to use collectd only applies to the cases where you have direct control over the software installed on an instance, as you do with Elastic Compute Cloud (EC2). Many services running on Amazon cloud can only be monitored via AWS CloudWatch. As a result, it is a common practice to use both the CloudWatch integration and the SignalFx collectd agent.

Irrespective of the mechanism by which you collect and send your metrics, you can take advantage of SignalFx’s importing of Amazon Web Services metadata, which applies not only to the relevant services, but can also be used with metrics collected via the SignalFx collectd agent. The metadata allows you to slice and dice by custom tags, availability zone, host names and other properties or dimensions.

Importing AWS CloudWatch metrics

If you chose to import AWS CloudWatch metrics when you added your Amazon Web Services integration (see Connect to AWS CloudWatch), metrics from CloudWatch will be synced into SignalFx.

Importing S3 metrics

By default, the daily storage metrics listed on this page are imported into SignalFx. If you also want to import the request metrics listed on that page, which are billed separately by Amazon, you must opt into them as described here and here.

Importing metrics via the Cloudwatch agent

Amazon provides a CloudWatch agent that lets you import more system-level metrics from Amazon EC2 instances and also lets you collect system-level metrics from on-premises servers, as described on this page. To import those metrics into SignalFx, add the namespace you used for the agent as a custom namespace in your Amazon Web Services integration (see Specifying Namespaces, Metrics, and Tags).

Importing AWS CloudWatch metric dimensions

SignalFx automatically imports relevant dimensions for each AWS CloudWatch metric. For example, if you are using Detailed Monitoring for EC2 instances, SignalFx imports the dimensions Auto ScalingGroupName, ImageId, InstanceId and InstanceType. These dimensions can be used to filter EC2 instance data. See Metrics Metadata for information about dimensions in SignalFx.

Unsupported characters within a dimension key are converted to underscores.

Importing account metadata and custom tags

For some services, SignalFx can apply or sync other Amazon Web Services metadata with metrics reported, allowing it to be used as filters or in group‑bys when visualizing metrics. It may take up to 15 minutes for metadata to be synced to your SignalFx data.

The following metadata is available for filtering metrics:

Custom Property Description
aws_account_id Amazon Web Services account ID under which the instance, volume or load balancer is running
aws_tag_[Name of tag] Custom tags applied to the instance, volume or load balancer

Account ID can be useful because SignalFx allows you to import metrics from more than one Amazon Web Services account. To distinguish between metrics from different accounts, the account ID is added as a property to the relevant metric time series.

For the supported services, SignalFx also syncs Amazon Web Services tags and makes them available as properties on metrics associated with the relevant resources, with the key aws_tag_[Name of tag]. For example, if an Amazon Web Services tag is named Production, it will be shown in SignalFx as aws_tag_Production.

The following sections provide details for the services that are currently supported.

Importing Auto Scaling metadata

For Auto Scaling, SignalFx will pull out properties of every group as well as any tags set on the group. For more information on these properties, see Amazon’s documentation.

Auto Scaling Name Custom Property Description
CreatedTime aws_created_time Time the resource was created at (e.g. Thu Apr 13 15:59:25 UTC 2017)
DefaulCoolDown aws_default_cool_down Amount of time, in seconds, after a scaling activity completes before another scaling activity can start
HealthCheckGracePeriod aws_health_check_grace_period Amount of time, in seconds, that Auto Scaling waits before checking the health status of an EC2 instance that has come into service
HealthCheckType aws_health_check_type Service to use for the health checks
LaunchConfigurationName aws_launch_configuration_name Name of the associated launch configuration
NewInstancesProtectedFromScaleIn aws_new_instances_protected_from_scale_in Indicates whether newly launched instances are protected from termination by Auto Scaling when scaling in
PlacementGroup aws_placement_group The name of the placement group into which you’ll launch your instances, if any
ServiceLinkedRoleARN aws_service_linked_role_arn ARN of the service-linked role that the Auto Scaling group uses to call other Amazon Web Services on your behalf
Stats aws_status Current state of the group when DeleteAuto ScalingGroup is in progress
VPCZoneIdentifier aws_vpc_zone_identifier One or more subnet IDs, if applicable, separated by commas
Region aws_region Amazon Web Services Region to which the Auto Scaling group belongs

Importing DynamoDB metadata

For DynamoDB, SignalFx will scan every table from your Amazon Web Services account and pull out properties of the table and any tags set on the table. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

DynamoDB Name Custom Property Description
ProvisionedThroughputDescription.ReadCapacityUnits aws_read_capacity_units Maximum number of strongly consistent reads consumed per second before DynamoDB returns a ThrottlingException
ProvisionedThroughputDescription.WriteCapcityUnits aws_write_capacity_units Maximum number of writes consumed per second before DynamoDB returns a ThrottlingException
TableName aws_table_name Name of the DynamoDB table
TableStatus aws_table_status Current state of the table

Importing EC2 metadata

For EC2, SignalFx will scan every instance ID from your Amazon Web Services account and pull out properties of the instance and any tags set on the instance.  Any called “Host” or “InstanceId” in SignalFx that matches the instance ID’s value, private DNS name, or private IP address will now have the same tags and properties of the instance ID.  Each instance property is prefixed with “aws_”. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

EC2 Name Custom Property Description
architecture aws_architecture Instance architecture (i386 or x86_64)
availability-zone aws_availability_zone The availability zone of the instance
dns-name aws_public_dns_name Public DNS name of the instance
hypervisor aws_hypervisor Hypervisor type of the instance (ovm or xen)
image-id aws_image_id ID of the image used to launch the instance
instance-id aws_instance_id ID of the instance
instance-state-name aws_state An object defining the state code and name of the instance
instance-type aws_instance_type Type of the instance
ip-address aws_public_ip_address The address of the Elastic IP address bound to the network interface
kernel-id aws_kernel_id Kernel ID
launch-time aws_launch_time The time when the instance was launched
private-dns-name aws_private_dns_name Private DNS name of the instance
reason aws_state_reason The state reason for the instance (if provided)
region aws_region The region in which the instance is running
reservation-id aws_reservation_id ID of the instance’s reservation
root-device-type aws_root_device_type Type of root device that the instance uses

Importing EC2 data for AWS Optimizer

SignalFx AWS Optimizer gives you actionable insight into cost-saving opportunities and underutilized EC2 investments. You can see usage patterns and cost attribution by InstanceType, AWS Region, AWS Availability Zone, as well as categories specific to your setup, such as Service, Team, or any other dimensions that are sourced from EC2 instance tags.

AWS Optimizer generates metrics from usage and cost data imported via calls to the AWS API. These generated metrics let you visualize and analyze EC2 usage and costs, as shown in built-in dashboards. You can also create detectors based on these metrics, so you can get alerted in real-time on unexpected changes in cost or usage patterns.

For the usage and cost data to be imported, make sure the following lines are in your Amazon Web Services Policy Document (for more information, see Connect to AWS CloudWatch):

"ec2:DescribeInstances",
"ec2:DescribeInstanceStatus",
"ec2:DescribeTags",
"ec2:DescribeReservedInstances",
"ec2:DescribeReservedInstancesModifications",
"organizations:DescribeOrganization",

Notes on using AWS Optimizer

  • AWS Optimizer is available in the SignalFx Enterprise Edition.
  • The imported data does not include Amazon Web Services billing data.
  • Data is not imported and metrics are not generated for EC2 Spot Instances.
  • If you have multiple AWS Accounts, you must add an Amazon Web Services integration for each account, and each integration must have “Import data for SignalFx AWS Optimizer” selected. If this is not the case, the generated metrics may not reflect accurate values.

Importing EBS metadata

For EBS, SignalFx will scan every volume ID from your Amazon Web Services account and pull out properties of the volume and any tags set on the volume. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

EBS Name Custom Property Description
availability-zone aws_availability_zone The Availability Zone in which the volume was created
create-time aws_create_time The time stamp when the volume was created
encrypted aws_encrypted The encryption status of the volume
iops aws_iops The number of I/O operations per second (IOPS) that the volume supports
kms_key_id aws_kms_key_id The full ARN of the Amazon Web Services customer master key used to protect the volume encryption key for the volume
size aws_size The size of the volume, in GiB
snapshot_id aws_snapshot_id The snapshot from which the volume was created
state aws_state The status of the volume
volume_id aws_volume_id The volume ID
volume_type aws_volume_type The Amazon EBS volume type

Importing ELB and Application ELB metadata

For ELB, SignalFx will scan every load balancer name from your Amazon Web Services account and pull out properties of the load balancer and any tags set on the load balancer. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

ELB Name Custom Property Description
create-time aws_create_time The time stamp when the load balancer was created

Importing ElastiCache metadata

For ElastiCache, SignalFx will scan every cluster and node from your Amazon Web Services account and pull out their properties as well as any tags set on the cluster or node. For more information on these properties, including acceptable values and constraints, see the Amazon Web Services documentation links in the table below.

ElastiCache Name Custom Property Description Applies to
ReplicationGroupId aws_replication_group_id The replication group to which this cluster belongs. If this field is empty, the cluster is not associated with any replication group. Cluster metrics that are part of a replication group
CacheClusterCreateTime aws_cache_cluster_create_time The date and time when the cluster was created. Cluster and node
Engine aws_engine The name of the cache engine to be used for this cluster. Cluster and node
EngineVersion aws_engine_version The version of the cache engine that is used in this cluster. Cluster and node
CustomerAvailabilityZone aws_availability_zone The Availability Zone where this node was created and now resides. Node only
CacheNodeCreateTime aws_cache_node_create_time The date and time when the cache node was created. Node only
n/a aws_cache_cluster_name Set to either the value of aws_replication_group_id (if applicable) or the value of the dimension CacheClusterId. * Cluster and node

* CacheClusterId is a dimension that is already on all ElastiCache metrics as retrieved from AWS Cloudwatch.

Importing RDS metadata

For RDS, SignalFx will scan every database instance from your Amazon Web Services account and pull out properties of each instance and any tags set on each instance. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

RDS Name Custom Property Description
AvailabilityZone aws_availability_zone Specifies the name of the Availability Zone the DB instance is located in.
DBClusterIdentifier aws_db_cluster_identifier If the DB instance is a member of a DB cluster, contains the name of the DB cluster that the DB instance is a member of.
DBInstanceClass aws_db_instance_class Contains the name of the compute and memory capacity class of the DB instance.
DBInstanceStatus aws_db_instance_status Specifies the current state of this database.
Engine aws_engine Provides the name of the database engine to be used for this DB instance.
EngineVersion aws_engine_version Indicates the database engine version.
InstanceCreateTime aws_instance_create_time Provides the date and time the DB instance was created.
Iops aws_iops Specifies the new Provisioned IOPS value for the DB instance that will be applied or is being applied.
MultiAZ aws_multi_az Specifies if the DB instance is a Multi-AZ deployment.
PubliclyAccessible aws_publicly_accessible Specifies the accessibility options for the DB instance. A value of true specifies an Internet-facing instance with a publicly resolvable DNS name, which resolves to a public IP address. A value of false specifies an internal instance with a DNS name that resolves to a private IP address.
ReadReplicaSourceDBInstanceIdentifier aws_read_replica_source_db_instance_identifier Contains the identifier of the source DB instance if this DB instance is a Read Replica.
SecondaryAvailabilityZone aws_second_availability_zone If present, specifies the name of the secondary Availability Zone for a DB instance with multi-AZ support.
StorageType aws_storage_type Specifies the storage type associated with the DB instance.

Importing AWS Lambda metadata

For AWS Lambda, SignalFx will scan every version of every function from your Amazon Web Services account and pull out properties of the function version and any tags set on the function. We will also look for any dimension called lambda_arn, which is the qualified ARN for an AWS Lambda function, and sync properties of the function version and any tags set on the function. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

AWS Lambda Filter Name Custom Property Description
CodeSha256 aws_function_code_sha256 The SHA256 hash of your function deployment package
CodeSize aws_function_code_size The size of the function .zip file you uploaded, in bytes
FunctionName aws_function_name The name of the function
MemorySize aws_function_memory_size The memory size you configured for the function, in MB
Runtime aws_function_runtime The runtime environment for the function
Timeout aws_function_timeout The function execution time at which Lambda should terminate the function
Version aws_function_version The version of the function
VpcConfig.vpcId aws_function_vpc_id The VPC ID associated with your function

For more information on monitoring AWS Lambda in SignalFx, see Monitor AWS Lambda.

Importing Redshift metadata

For RedShift, SignalFx will scan every cluster from your Amazon Web Services account and pull out properties of the cluster and any tags set on the cluster. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

Redshift Name Custom Property Description
ClusterIdentifier aws_cluster_identifier The unique identifier of the cluster.
AvailabilityZone aws_availability_zone The name of the Availability Zone in which the cluster is located.
ClusterCreateTime aws_cluster_create_time The date and time that the cluster was created.
ClusterStatus aws_cluster_status The current state of the cluster.
ClusterRevisionNumber aws_cluster_revision_number The specific revision number of the database in the cluster.
ClusterVersion aws_cluster_version The version ID of the Amazon Redshift engine that is running on the cluster.
NodeType aws_cluster_node_type The node type for the nodes in the cluster.
DBName aws_cluster_db_name The name of the initial database that was created when the cluster was created.
Encrypted aws_cluster_encrypted A Boolean value that, if true, indicates that data in the cluster is encrypted at rest.
MasterUsername aws_cluster_master_username The master user name for the cluster. This name is used to connect to the database that is specified in the DBName parameter.
PubliclyAccessible aws_cluster_publicly_accessible A Boolean value that, if true, indicates that the cluster can be accessed from a public network.

Importing SQS metadata

For SQS, SignalFx will pull out properties of every queue as well as any tags set on the queue. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

SQS Name Custom Property Description
QueueArn aws_queue_arn Amazon Web Services resource name of the SQS queue
QueueURL aws_queue_url URL for the SQS queue
MaximumMessageSize aws_maximum_message_size Maximum size of a message that SQS will accept, in bytes; larger messages will be rejected
CreateTimestamp aws_created_timestamp Time that the SQS queue was created
VisibilityTimeout aws_visibility_timeout The visibility timeout for the queue
FifoQueue aws_fifo_queue Indicates whether the queue is a fifo queue
Region aws_region The region in which the SQS resides

Using CloudWatch metrics in SignalFx

AWS CloudWatch statistics

Much like SignalFx, AWS CloudWatch uses rollups to summarize metrics, and it refers to them as “statistics”. However, because there is not a one-for-one mapping to SignalFx’s data model, the CloudWatch rollups are not directly accessible through the rollup selection menu in the Chart Builder. Instead, they are captured as individual time series through the use of the dimension stat.

AWS Statistic SignalFx dimension Definition
Average stat:mean Mean value of metric over the sampling period
Maximum stat:upper Maximum value of metric over the sampling period
Minimum stat:lower Minimum value of metric over the sampling period
Data Samples stat:count Number of samples over the sampling period
Sum stat:sum Sum of all values that occurred over the sampling period

In other words, to use an AWS CloudWatch metric in a plot, you must always specify the metric name along with a filter for stat that is appropriate to the metric you have chosen. For example, if you are using the metric NetworkPacketsIn, per the Amazon Web Services CloudWatch documentation for EC2 metrics, the only statistics that are meaningful are Minimum, Maximum and Average, so you should choose the dimension stat with a value of either lower, upper or mean, respectively, depending on which statistic you want to use.

Tip

If you see “Rollup: Multiple” in the plot line where you have selected a CloudWatch metric, then you haven’t yet specified which statistic you want to use, and you should do so immediately.

When syncing data from CloudWatch to SignalFx, a 60-second sampling period is used. See AWS CloudWatch documentation for more detailed information.

Amazon Web Services namespaces

SignalFx imports the namespace for Amazon Web Services using the dimension namespace. For most services, the namespaces take the format of “AWS/<name_of_service>”, e.g. “AWS/EC2” or “AWS/ELB”. This distinction is important when you want to use metrics that have the same name across services, such as CPUUtilization, but only for a single service (say, EC2 and not ECS).

To control the amount of data being imported, you can specify which namespaces you want to import, as well as what data to import or exclude from each namespace. For more information, see Specifying Namespaces, Metrics, and Tags.

Uniquely identifying Amazon Web Services instances

The Amazon Web Services instance ID is not a unique identifier. To uniquely identify an AWS instance, concatenate instanceId, region, and accountID separated by underscores “_”, as follows:

instanceId_region_accountID

We’ve provided a script that you can use to print a unique ID for an Amazon Web Services instance, suitable for use in SignalFx. Find the script here.

To construct the identifier manually, first obtain metadata values for each instance from the following URL:

curl http://169.254.169.254/latest/dynamic/instance-identity/document

Here’s an example response from that URL:

{
"devpayProductCodes" : null,
 "privateIp" : "10.1.15.204",
 "availabilityZone" : "us-east-1a",
 "version" : "2010-08-31",
 "accountId" : "134183635603",
 "instanceId" : "i-a99f9802",
 "billingProducts" : null,
 "instanceType" : "c3.2xlarge",
 "pendingTime" : "2015-09-02T16:45:40Z",
 "imageId" : "ami-2ef44746",
 "kernelId" : null,
 "ramdiskId" : null,
 "architecture" : "x86_64",
 "region" : "us-east-1"
}

Parse the response to obtain values for Amazon Web Services instanceId, region, and accountId. Finally, concatenate them separated by underscores “_” as follows: instanceId_region_accountId. Use this identifier as the value for “sfxdim_AWSUniqueId” in the procedure shown above.