Docs » Integrations Guide » Monitor Amazon Web Services

Monitor Amazon Web Services 🔗

If you’re an Amazon Web Services customer, SignalFx can automatically import metrics and metadata about your services from AWS CloudWatch. If you haven’t already done so, follow the instructions below to connect SignalFx to AWS.

Connect to AWS CloudWatch 🔗

You must be an administrator of your SignalFx account to connect SignalFx to AWS CloudWatch. The first time you connect SignalFx to CloudWatch, you will:

  • Create a new policy and IAM role in AWS for SignalFx to use.
  • Provide information from SignalFx to the new role.
  • Provide SignalFx with the role ARN.

Subsequent connections to CloudWatch can use the same policy, or you can create a different one. Either way, you must attach the policy with the role you create.

Part 1 - Create a policy in AWS 🔗

If this is the first time you have connected to CloudWatch, or if you want to create a new policy, follow the steps below. If you have already installed at least one AWS integration and want to use the same policy, you can skip to Part 2 - Connect AWS to SignalFx.

  1. To get started, log into your Amazon Web Services account.

  2. From the Services list, select IAM to open Identity & Access Management.

  3. In the sidebar, click Policies, then click Create Policy.

  4. Click the JSON tab. You may see a message saying the policy is not valid; dismiss the message. Then delete the existing text in the policy.

    ../_images/aws-json-tab.png
  5. Copy the following code and paste it into the policy:

    {
     "Version": "2012-10-17",
     "Statement": [
      {
       "Effect": "Allow",
       "Action": [
        "dynamodb:ListTables",
        "dynamodb:DescribeTable",
        "dynamodb:ListTagsOfResource",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceStatus",
        "ec2:DescribeVolumes",
        "ec2:DescribeReservedInstances",
        "ec2:DescribeReservedInstancesModifications",
        "ec2:DescribeTags",
        "organizations:DescribeOrganization",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:DescribeAlarms",
        "sqs:ListQueues",
        "sqs:GetQueueAttributes",
        "sqs:ListQueueTags",
        "elasticmapreduce:ListClusters",
        "elasticmapreduce:DescribeCluster",
        "kinesis:ListShards",
        "kinesis:ListStreams",
        "kinesis:DescribeStream",
        "kinesis:ListTagsForStream",
        "rds:DescribeDBInstances",
        "rds:ListTagsForResource",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeTags",
        "elasticache:describeCacheClusters",
        "redshift:DescribeClusters",
        "lambda:GetAlias",
        "lambda:ListFunctions",
        "lambda:ListTags",
        "autoscaling:DescribeAutoScalingGroups",
        "s3:ListAllMyBuckets",
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:GetBucketTagging",
        "ecs:ListServices",
        "ecs:ListTasks",
        "ecs:DescribeTasks",
        "ecs:DescribeServices",
        "ecs:ListClusters",
        "ecs:DescribeClusters",
        "ecs:ListTaskDefinitions",
        "ecs:ListTagsForResource",
        "apigateway:GET",
        "cloudfront:ListDistributions",
        "cloudfront:ListTagsForResource",
        "tag:GetResources",
        "es:ListDomainNames",
        "es:DescribeElasticsearchDomain"
       ],
       "Resource": "*"
      }
     ]
    }
    

    If the invalid policy message appears again, confirm you have pasted the entire text correctly.

  6. Click Review policy.

  7. Give the policy a name and optional description, then click Create policy.

  8. If you are installing an integration for GovCloud or China, skip to AWS Authentication - Security Token method. Otherwise, continue with Part 2 - Connect AWS to SignalFx below.

Part 2 - Connect AWS to SignalFx 🔗

Note

If you are installing an integration for GovCloud or China, do not perform the steps below. Instead, skip to AWS Authentication - Security Token method.

  1. In the Amazon Web Services console, if the Identity & Access Management page is not open, display the Services list and select IAM.

  2. Select Roles in the sidebar, then click Create role. For type of trusted entity, select Another AWS account, then select Require external ID to display the External ID field.

    ../_images/aws-create-role-1.png
  3. Next, you’ll provide the role with information from SignalFx. In a new tab, open SignalFx and click Integrations to open the Integrations page. Click the Amazon Web Services tile. Display the Setup tab, then click New Integration.

  4. Give the integration a name, and then, from the fields that appear, copy and paste the SignalFx Account ID and External ID from the SignalFx tab (first illustration below) into the Account ID and External ID fields in the AWS tab (second illustration below). (You will return to the SignalFx tab in a later step, so leave it open). Then click Next: Permissions.

    ../_images/signalfx-account-id.png


    ../_images/aws-create-role-2.png
  5. Find the policy that you created earlier in Part 1 - Create a policy in AWS, select it (click the checkbox to its left) and then click Next: Tags.

  6. Add tags to the role, if desired. You can use tags to specify the resources for which data will be imported or excluded. Click Next: Review.

  7. Give the role a name and optional description, review the information that you’ve entered, then click Create Role. You have now created a role that SignalFx can use to connect to AWS CloudWatch.

Part 3 - Complete the integration in SignalFx 🔗

  1. Select Roles in the sidebar. Click on the role you just created, and copy the Role ARN.

    ../_images/aws-role-arn-1.png
  2. Switch back to the SignalFx tab, and paste the ARN in the text field labelled Role ARN.

    ../_images/aws-save-and-enable.png
  3. By default, SignalFx will import all available data for every CloudWatch-enabled region in your Amazon Web Services account, excluding GovCloud and China regions. To import only a subset, or to import from those regions, click All Regions to display options for choosing specific regions.

    Each AWS integration can import data from either GovCloud regions, China regions, or other regions. For example, if you want to import data from US East regions and from GovCloud (US‑East), you will have to add two AWS integrations.

  4. Select the data that you wish SignalFx to import from Amazon Web Services. Except for the first option below, you can change this configuration later.

    • “Synchronize metadata”: SignalFx imports the properties of your AWS hosts and services, including region, account ID, and more. You can use this information to search, filter, and aggregate your metrics.
    • “Import data for AWS Optimizer”: With this option selected, SignalFx will import cost and usage data from AWS and will also generate metrics you can use to visualize EC2 usage and approximated costs, as well as get alerted in real-time on unexpected changes in cost or usage.

    Notes on using AWS Optimizer

    • AWS Optimizer is available in the SignalFx Enterprise Edition.
    • The imported data does not include Amazon Web Services billing data.
    • Data is not imported and metrics are not generated for EC2 Spot Instances.
    • If you have multiple AWS Accounts, you must add an Amazon Web Services integration for each account, and each integration must have “Import data for SignalFx AWS Optimizer” selected. If this is not the case, the generated metrics may not reflect accurate values.
    • “Import CloudWatch”: With this option selected, SignalFx will import CloudWatch metrics.
      • By default, SignalFx imports all metrics from all resources in all built-in namespaces. You can limit what data is imported on a per-namespace basis, and you can also import data from custom namespaces. For more information, see Specifying Namespaces, Metrics, and Tags below.
      • See Importing S3 metrics and Importing metrics via the Cloudwatch agent for specifics on importing data from those sources.
      • You can set the poll rate for importing CloudWatch metrics to 5 minutes or 1 minute; the latter is appropriate if you are using Detailed Monitoring on an Amazon Web Services namespace.
  5. Click Save and Enable.

    SignalFx will begin to validate the integration. When you see the Validated! message, your Security Token is verified, and SignalFx begins to receive metrics from AWS.

Tips

To create additional integrations (for example, to monitor different regions), repeat Part 2 - Connect AWS to SignalFx and Part 3 - Complete the integration in SignalFx for each integration.

You can use the Actions menu for an integration to disable or enable it, or to delete it.

Your CloudWatch integration is now complete.

If you installed this integration while going through the Quick Start guide, continue by installing the Smart Agent, which monitors host infrastructure metrics.

Otherwise, use the information below to learn more about working with your AWS data in SignalFx.

AWS integration overview 🔗

SignalFx provides a robust integration with CloudWatch, has a CloudWatch-powered mode for the Infrastructure Navigator, and includes many built-in-dashboards to help you get started monitoring Amazon Web Services.

You can also monitor Amazon Web Services instances and the services running on them by using the SignalFx Smart Agent. The SignalFx Smart Agent offers a much higher degree of customization than is possible with AWS CloudWatch, and may be preferable for instances where you want to see metrics at a finer resolution, or where detailed control over the metrics sent matters.

Note

The option to use the Smart Agent only applies to the cases where you have direct control over the software installed on an instance, as you do with Elastic Compute Cloud (EC2). Many services running on Amazon cloud can only be monitored via AWS CloudWatch. As a result, it is a common practice to use both the CloudWatch integration and the SignalFx Smart Agent.

Irrespective of the mechanism by which you collect and send your metrics, you can take advantage of SignalFx’s importing of Amazon Web Services metadata, which applies not only to the relevant services, but can also be used with metrics collected via the SignalFx collectd agent. The metadata allows you to slice and dice by custom tags, availability zone, host names and other properties or dimensions.

Specifying Namespaces, Metrics, and Tags 🔗

By default, SignalFx imports all metrics from all resources in all built-in namespaces. If you want to limit the amount of data AWS sends to SignalFx, you can specify the built-in namespaces from which you want to receive data. For each namespace, you can then filter the data you import based on tags on the resource and/or by metric name. You can also import data from custom namespaces.

You must be an administrator of your SignalFx account to set these options.

../_images/cloudwatch-01.png
  • To specify built-in namespaces, click Select namespaces, then choose the namespace(s) for which you want data.
  • To specify custom namespaces, click Add custom namespaces, type the name of a custom namespace, then press Enter. You can specify multiple custom namespaces.

Specifying a filter on data to import has no effect on tag syncing. SignalFx will continue to sync all tags for all resources in a specified namespace.

In the example below, we are importing data only from Amazon ElasticSearch Service and EC2. By default, Import All is selected, which means SignalFx will import all metrics from all resources in both namespaces.

../_images/cloudwatch-02.png

We are going to limit the data coming in from EC2. Click the dropdown arrow to see options for including or excluding data.

../_images/cloudwatch-03.png
  • Use Import only if you want to specify a filter for the data to import (whitelist); no other data will be imported.
  • Use Don’t import if you want to specify a filter for the data to exclude (blacklist); all other data will be imported.

Tags are used to specify the resources for which data will be imported or excluded. In the example below, we are excluding one tag, which means SignalFx will import all metrics from all resources that do not have the tag version:canary.

../_images/cloudwatch-04.png

SignalFx adds the prefix aws_tag_ to the names of tags after we sync them from AWS, to indicate their origin. For example, an AWS tag version:canary would appear in SignalFx as aws_tag_version:canary. When you filter your CloudWatch integration by tag, enter the name of the tag as it appears in AWS.

You can also choose specific metrics to import or exclude. For example, consider the following conditions.

../_images/aws-metric-tag.png

Only metricA and metricB will be imported, and only for resources specified by the tags. In other words:

  • For a resource that has the tag env:prod or env:beta, metricA and metricB will be imported.
  • For a resource that does not have either the tag env:prod or env:beta, no metrics will be imported.
  • No other metrics will be imported.

Wildcards are supported. For example, if you want to import a metric, but only for a resource that has specified tags with any value, you could use a condition like the following.

../_images/aws-metric-tag-wildcard.png

In this case, MetricA and MetricB will be imported for resources that have the env tag set to any value. No other metrics will be imported.

You can use the Actions menu to the right of a specified namespace to copy or paste filters from one namespace to another, to clear the filters (for example, if you want to specify different metrics or tags), or to remove the namespace from the list. Removing a namespace means that SignalFx will no longer import metrics from that namespace.

When you have finished specifying the namespaces, metrics, and tags to include or exclude, click Save.

Note

It is possible to specify more complex filtering options for a namespace through the use of the SignalFx API. In this case, instead of seeing a visual representation of the specified metrics or tags, you will see a message indicating that the filter was defined programmatically. Click View filter code to see which metrics and tags are being included or excluded for that namespace.

Importing AWS CloudWatch metrics 🔗

If you chose to import AWS CloudWatch metrics when you added your Amazon Web Services integration (see Connect to AWS CloudWatch), metrics from CloudWatch will be synced into SignalFx.

Importing S3 metrics 🔗

By default, the daily storage metrics listed on this page are imported into SignalFx. If you also want to import the request metrics listed on that page, which are billed separately by Amazon, you must opt into them as described here and here.

See also Importing S3 metadata.

Importing metrics via the Cloudwatch agent 🔗

Amazon provides a CloudWatch agent that lets you import more system-level metrics from Amazon EC2 instances and also lets you collect system-level metrics from on-premises servers, as described on this page. To import those metrics into SignalFx, add the namespace you used for the agent as a custom namespace in your Amazon Web Services integration (see Specifying Namespaces, Metrics, and Tags).

Importing AWS CloudWatch metric dimensions 🔗

SignalFx automatically imports relevant dimensions for each AWS CloudWatch metric. For example, if you are using Detailed Monitoring for EC2 instances, SignalFx imports the dimensions AutoScalingGroupName, ImageId, InstanceId and InstanceType. These dimensions can be used to filter EC2 instance data. See Metric Metadata for information about dimensions in SignalFx.

Unsupported characters within a dimension key are converted to underscores.

Importing account metadata and custom tags 🔗

For some services, SignalFx can apply or sync other Amazon Web Services metadata with metrics reported, allowing it to be used as filters or in group‑bys when visualizing metrics. It may take up to 15 minutes for metadata to be synced to your SignalFx data.

Note

Filtering using tags is valid only in namespaces for which SignalFx syncs tags. For more information, see Amazon Web Services namespaces.

The following metadata is available for filtering metrics:

Custom Property Description
aws_account_id Amazon Web Services account ID under which the instance, volume or load balancer is running
aws_tag_[Name of tag] Custom tags applied to the instance, volume or load balancer

Account ID can be useful because SignalFx allows you to import metrics from more than one Amazon Web Services account. To distinguish between metrics from different accounts, the account ID is added as a property to the relevant metric time series.

For the supported services, SignalFx also syncs Amazon Web Services tags and makes them available as properties on metrics associated with the relevant resources, with the key aws_tag_[Name of tag]. For example, if an Amazon Web Services tag is named Production, it will be shown in SignalFx as aws_tag_Production.

The following sections provide details for the services that are currently supported.

Importing API Gateway metadata 🔗

For API Gateway, SignalFx will pull out the names and tags of every REST API and stage. For more information, see Amazon’s documentation.

API Gateway Name Custom Property Description
ApiName aws_rest_api_name The API’s name
Stage aws_stage_name The first path segment in the Uniform Resource Identifier (URI) of a call to API Gateway

Importing Auto Scaling metadata 🔗

For Auto Scaling, SignalFx will pull out properties of every group as well as any tags set on the group. For more information on these properties, see Amazon’s documentation.

Auto Scaling Name Custom Property Description
CreatedTime aws_created_time Time the resource was created at (e.g. Thu Apr 13 15:59:25 UTC 2017)
DefaultCoolDown aws_default_cool_down Amount of time, in seconds, after a scaling activity completes before another scaling activity can start
HealthCheckGracePeriod aws_health_check_grace_period Amount of time, in seconds, that Auto Scaling waits before checking the health status of an EC2 instance that has come into service
HealthCheckType aws_health_check_type Service to use for the health checks
LaunchConfigurationName aws_launch_configuration_name Name of the associated launch configuration
NewInstancesProtectedFromScaleIn aws_new_instances_protected_from_scale_in Indicates whether newly launched instances are protected from termination by Auto Scaling when scaling in
PlacementGroup aws_placement_group The name of the placement group into which you’ll launch your instances, if any
ServiceLinkedRoleARN aws_service_linked_role_arn ARN of the service-linked role that the Auto Scaling group uses to call other Amazon Web Services on your behalf
Stats aws_status Current state of the group when DeleteAuto ScalingGroup is in progress
VPCZoneIdentifier aws_vpc_zone_identifier One or more subnet IDs, if applicable, separated by commas
Region aws_region Amazon Web Services Region to which the Auto Scaling group belongs

Importing CloudFront metadata 🔗

For CloudFront, SignalFx will scan every distribution from your Amazon Web Services account and pull out properties of the distribution and any tags set on the distribution. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

CloudFront Name Custom Property Description
Id aws_distribution_id The identifier for the distribution, for example EDFDVBD632BHDS5.
DomainName aws_domain_name The domain name corresponding to the distribution, for example d111111abcdef8.cloudfront.net.

Importing DynamoDB metadata 🔗

For DynamoDB, SignalFx will scan every table from your Amazon Web Services account and pull out properties of the table and any tags set on the table. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

DynamoDB Name Custom Property Description
ProvisionedThroughputDescription.ReadCapacityUnits aws_read_capacity_units Maximum number of strongly consistent reads consumed per second before DynamoDB returns a ThrottlingException
ProvisionedThroughputDescription.WriteCapcityUnits aws_write_capacity_units Maximum number of writes consumed per second before DynamoDB returns a ThrottlingException
TableName aws_table_name Name of the DynamoDB table
TableStatus aws_table_status Current state of the table

Importing Elasticsearch metadata 🔗

For Elasticsearch, SignalFx will scan every domain from your Amazon Web Services account and pull out the version as well as any tags set on the domain. For more information, see Amazon’s documentation.

Elasticsearch Name Custom Property Description
ElasticsearchVersion aws_es_version The Elasticsearch version, for example 7.1.

Importing EBS metadata 🔗

For EBS, SignalFx will scan every volume ID from your Amazon Web Services account and pull out properties of the volume and any tags set on the volume. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

EBS Name Custom Property Description
attachment_state aws_attachment_state The attachment state of the volume
availability-zone aws_availability_zone The Availability Zone in which the volume was created
create-time aws_create_time The time stamp when the volume was created
delete_on_termination aws_delete_on_termination Whether or not a volume will be deleted if the instance it is attached to is terminated
encrypted aws_encrypted The encryption status of the volume
instance_id aws_instance_id ID of the instance to which the volume is attached. This property will be propagated only if the volume is attached to an instance
iops aws_iops The number of I/O operations per second (IOPS) that the volume supports
kms_key_id aws_kms_key_id The full ARN of the Amazon Web Services customer master key used to protect the volume encryption key for the volume
size aws_size The size of the volume, in GiB
snapshot_id aws_snapshot_id The snapshot from which the volume was created
state aws_state The status of the volume
volume_id aws_volume_id The volume ID
volume_type aws_volume_type The Amazon EBS volume type

Importing EC2 metadata 🔗

For EC2, SignalFx will scan every instance ID from your Amazon Web Services account and pull out properties of the instance and any tags set on the instance.  Any called “Host” or “InstanceId” in SignalFx that matches the instance ID’s value, private DNS name, or private IP address will now have the same tags and properties of the instance ID.  Each instance property is prefixed with ”aws_”. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

EC2 Name Custom Property Description
architecture aws_architecture Instance architecture (i386 or x86_64)
availability-zone aws_availability_zone The availability zone of the instance
dns-name aws_public_dns_name Public DNS name of the instance
hypervisor aws_hypervisor Hypervisor type of the instance (ovm or xen)
image-id aws_image_id ID of the image used to launch the instance
instance-id aws_instance_id ID of the instance
instance-state-name aws_state An object defining the state code and name of the instance
instance-type aws_instance_type Type of the instance
ip-address aws_public_ip_address The address of the Elastic IP address bound to the network interface
kernel-id aws_kernel_id Kernel ID
launch-time aws_launch_time The time when the instance was launched
private-dns-name aws_private_dns_name Private DNS name of the instance
reason aws_state_reason The state reason for the instance (if provided)
region aws_region The region in which the instance is running
reservation-id aws_reservation_id ID of the instance’s reservation
root-device-type aws_root_device_type Type of root device that the instance uses

Importing EC2 data for AWS Optimizer 🔗

SignalFx AWS Optimizer gives you actionable insight into cost-saving opportunities and underutilized EC2 investments. You can see usage patterns and cost attribution by InstanceType, AWS Region, AWS Availability Zone, as well as categories specific to your setup, such as Service, Team, or any other dimensions that are sourced from EC2 instance tags.

AWS Optimizer generates metrics from usage and cost data imported via calls to the AWS API. These generated metrics let you visualize and analyze EC2 usage and costs, as shown in built-in dashboards. You can also create detectors based on these metrics, so you can get alerted in real-time on unexpected changes in cost or usage patterns.

For the usage and cost data to be imported, make sure the following lines are in your Amazon Web Services Policy Document (for more information, see Connect to AWS CloudWatch):

"ec2:DescribeInstances",
"ec2:DescribeInstanceStatus",
"ec2:DescribeTags",
"ec2:DescribeReservedInstances",
"ec2:DescribeReservedInstancesModifications",
"organizations:DescribeOrganization",

Notes on using AWS Optimizer

  • AWS Optimizer is available in the SignalFx Enterprise Edition.
  • The imported data does not include Amazon Web Services billing data.
  • Data is not imported and metrics are not generated for EC2 Spot Instances.
  • If you have multiple AWS Accounts, you must add an Amazon Web Services integration for each account, and each integration must have “Import data for SignalFx AWS Optimizer” selected. If this is not the case, the generated metrics may not reflect accurate values.

Importing Elastic Container Service (ECS) metadata 🔗

For ECS, SignalFx will scan every cluster and service from your Amazon Web Services account and pull out their properties as well as any tags set on the cluster or service. For more information on these properties see Amazon’s documentation.

ECS Name Custom Property Description
ClusterName aws_cluster_name A user-generated string that you use to identify your cluster.
ServiceName aws_service_name The name of your service.

Importing Classic, Application, and Network ELB metadata 🔗

For ELB, SignalFx will scan every load balancer name from your Amazon Web Services account and pull out properties of the load balancer and any tags set on the load balancer. For more information on these properties, including acceptable values and constraints, see Amazon documentation for Classic ELBs and Amazon documentation for Application/Network ELBs.

ELB Name Custom Property Description
create-time aws_create_time The time stamp when the load balancer was created

Importing ElastiCache metadata 🔗

For ElastiCache, SignalFx will scan every cluster and node from your Amazon Web Services account and pull out their properties as well as any tags set on the cluster or node. For more information on these properties, including acceptable values and constraints, see the Amazon Web Services documentation links in the table below.

ElastiCache Name Custom Property Description Applies to
ReplicationGroupId aws_replication_group_id The replication group to which this cluster belongs. If this field is empty, the cluster is not associated with any replication group. Cluster metrics that are part of a replication group
CacheClusterCreateTime aws_cache_cluster_create_time The date and time when the cluster was created. Cluster and node
Engine aws_engine The name of the cache engine to be used for this cluster. Cluster and node
EngineVersion aws_engine_version The version of the cache engine that is used in this cluster. Cluster and node
CustomerAvailabilityZone aws_availability_zone The Availability Zone where this node was created and now resides. Node only
CacheNodeCreateTime aws_cache_node_create_time The date and time when the cache node was created. Node only
n/a aws_cache_cluster_name Set to either the value of aws_replication_group_id (if applicable) or the value of the dimension CacheClusterId. * Cluster and node

* CacheClusterId is a dimension that is already on all ElastiCache metrics as retrieved from AWS Cloudwatch.

Importing EMR metadata 🔗

For EMR, SignalFx will pull out properties of every cluster as well as any tags set on each cluster. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

EMR Name Custom Property Description
Id aws_cluster_id Amazon Web Services identifier of the cluster
Name aws_cluster_name The name you gave the cluster
AutoScalingRole aws_auto_scaling_role An IAM role for automatic scaling policies
CustomAmiId aws_custom_ami_id The ID of a custom Amazon EBS-backed Linux AMI if the cluster uses a custom AMI
InstanceCollectionType aws_instance_collection_type The instance group configuration of the cluster
LogUri aws_log_uri The path to the Amazon S3 location where logs for this cluster are stored
MasterPublicDnsName aws_master_public_dns_name The DNS name of the master node
ReleaseLabel aws_release_label The Amazon EMR release label, which determines the version of open-source application packages installed on the cluster
RepoUpgradeOnBoot aws_repo_upgrade_on_boot Applies only when CustomAmiID is used
RequestedAmiVersion aws_requested_ami_version The AMI version requested for this cluster
RunningAmiVersion aws_running_ami_version The AMI version running on this cluster
ScaleDownBehavior aws_scale_down_behavior The way that individual Amazon EC2 instances terminate when an automatic scale-in activity occurs or an instance group is resized
SecurityConfiguration aws_security_configuration The name of the security configuration applied to the cluster
ServiceRole aws_service_role The IAM role that will be assumed by the Amazon EMR service to access AWS resources on your behalf
Status aws_status The current status details about the cluster
AutoTerminate aws_auto_terminate Specifies whether the cluster should terminate after completing all steps
TerminationProtected aws_termination_protected Indicates whether Amazon EMR will lock the cluster to prevent the EC2 instances from being terminated by an API call or user intervention, or in the event of a cluster error
VisibleToAllUsers aws_visible_to_all_users Indicates whether the cluster is visible to all IAM users of the AWS account associated with the cluster
NormalizedInstanceHours aws_normalized_instance_hours An approximation of the cost of the cluster, represented in m1.small/hours

Importing Kinesis Streams metadata 🔗

For Kinesis Streams, SignalFx will pull out properties of every stream as well as any tags set on each stream. If shard-level metrics are enabled in AWS, SignalFx will also apply properties and tags to Kinesis shards for their respective parent streams. For more information, see Amazon’s documentation.

Kinesis Name Custom Property Description
StreamName aws_stream_name The name of the stream
StreamStatus aws_stream_status The server-side encryption type used on the stream
RetentionPeriodHours aws_retention_period_hours The current retention period, in hours

Importing RDS metadata 🔗

For RDS, SignalFx will scan every database instance from your Amazon Web Services account and pull out properties of each instance and any tags set on each instance. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

RDS Name Custom Property Description
AvailabilityZone aws_availability_zone Specifies the name of the Availability Zone the DB instance is located in.
DBClusterIdentifier aws_db_cluster_identifier If the DB instance is a member of a DB cluster, contains the name of the DB cluster that the DB instance is a member of.
DBInstanceClass aws_db_instance_class Contains the name of the compute and memory capacity class of the DB instance.
DBInstanceStatus aws_db_instance_status Specifies the current state of this database.
Engine aws_engine Provides the name of the database engine to be used for this DB instance.
EngineVersion aws_engine_version Indicates the database engine version.
InstanceCreateTime aws_instance_create_time Provides the date and time the DB instance was created.
Iops aws_iops Specifies the new Provisioned IOPS value for the DB instance that will be applied or is being applied.
MultiAZ aws_multi_az Specifies if the DB instance is a Multi-AZ deployment.
PubliclyAccessible aws_publicly_accessible Specifies the accessibility options for the DB instance. A value of true specifies an Internet-facing instance with a publicly resolvable DNS name, which resolves to a public IP address. A value of false specifies an internal instance with a DNS name that resolves to a private IP address.
ReadReplicaSourceDBInstanceIdentifier aws_read_replica_source_db_instance_identifier Contains the identifier of the source DB instance if this DB instance is a Read Replica.
SecondaryAvailabilityZone aws_second_availability_zone If present, specifies the name of the secondary Availability Zone for a DB instance with multi-AZ support.
StorageType aws_storage_type Specifies the storage type associated with the DB instance.

Importing AWS Lambda metadata 🔗

For AWS Lambda, SignalFx will scan every version of every function from your Amazon Web Services account and pull out properties of the function version and any tags set on the function. We will also look for any dimension called lambda_arn, which is the qualified ARN for an AWS Lambda function, and sync properties of the function version and any tags set on the function. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

AWS Lambda Filter Name Custom Property Description
CodeSha256 aws_function_code_sha256 The SHA256 hash of your function deployment package
CodeSize aws_function_code_size The size of the function .zip file you uploaded, in bytes
FunctionName aws_function_name The name of the function
MemorySize aws_function_memory_size The memory size you configured for the function, in MB
Runtime aws_function_runtime The runtime environment for the function
Timeout aws_function_timeout The function execution time at which Lambda should terminate the function
Version aws_function_version The version of the function
VpcConfig.vpcId aws_function_vpc_id The VPC ID associated with your function

For more information on monitoring AWS Lambda in SignalFx, see Monitor AWS Lambda.

Importing Redshift metadata 🔗

For RedShift, SignalFx will scan every cluster from your Amazon Web Services account and pull out properties of the cluster and any tags set on the cluster. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

Redshift Name Custom Property Description
ClusterIdentifier aws_cluster_identifier The unique identifier of the cluster.
AvailabilityZone aws_availability_zone The name of the Availability Zone in which the cluster is located.
ClusterCreateTime aws_cluster_create_time The date and time that the cluster was created.
ClusterStatus aws_cluster_status The current state of the cluster.
ClusterRevisionNumber aws_cluster_revision_number The specific revision number of the database in the cluster.
ClusterVersion aws_cluster_version The version ID of the Amazon Redshift engine that is running on the cluster.
NodeType aws_cluster_node_type The node type for the nodes in the cluster.
DBName aws_cluster_db_name The name of the initial database that was created when the cluster was created.
Encrypted aws_cluster_encrypted A Boolean value that, if true, indicates that data in the cluster is encrypted at rest.
MasterUsername aws_cluster_master_username The master user name for the cluster. This name is used to connect to the database that is specified in the DBName parameter.
PubliclyAccessible aws_cluster_publicly_accessible A Boolean value that, if true, indicates that the cluster can be accessed from a public network.

Importing SQS metadata 🔗

For SQS, SignalFx will pull out properties of every queue as well as any tags set on the queue. For more information on these properties, including acceptable values and constraints, see Amazon’s documentation.

SQS Name Custom Property Description
QueueArn aws_queue_arn Amazon Web Services resource name of the SQS queue
QueueURL aws_queue_url URL for the SQS queue
MaximumMessageSize aws_maximum_message_size Maximum size of a message that SQS will accept, in bytes; larger messages will be rejected
CreateTimestamp aws_created_timestamp Time that the SQS queue was created
VisibilityTimeout aws_visibility_timeout The visibility timeout for the queue
FifoQueue aws_fifo_queue Indicates whether the queue is a fifo queue
Region aws_region The region in which the SQS resides

Importing S3 metadata 🔗

For S3, SignalFx will pull out the region in which the bucket resides, as well as any tags set on buckets. Metadata will be pulled out only for non-empty buckets. For more information on S3 bucket tagging, see Amazon’s documentation.

S3 Name Custom Property Description
Region aws_region The region in which the S3 bucket resides

Using CloudWatch metrics in SignalFx 🔗

AWS CloudWatch statistics 🔗

Much like SignalFx, AWS CloudWatch uses rollups to summarize metrics, and it refers to them as “statistics”. However, because there is not a one-for-one mapping to SignalFx’s data model, the CloudWatch rollups are not directly accessible through the rollup selection menu in the Chart Builder. Instead, they are captured as individual time series through the use of the dimension stat.

AWS Statistic SignalFx dimension Definition
Average stat:mean Mean value of metric over the sampling period
Maximum stat:upper Maximum value of metric over the sampling period
Minimum stat:lower Minimum value of metric over the sampling period
Data Samples stat:count Number of samples over the sampling period
Sum stat:sum Sum of all values that occurred over the sampling period

In other words, to use an AWS CloudWatch metric in a plot, you must always specify the metric name along with a filter for stat that is appropriate to the metric you have chosen. For example, if you are using the metric NetworkPacketsIn, per the Amazon Web Services CloudWatch documentation for EC2 metrics, the only statistics that are meaningful are Minimum, Maximum and Average, so you should choose the dimension stat with a value of either lower, upper or mean, respectively, depending on which statistic you want to use.

Tip

If you see “Rollup: Multiple” in the plot line where you have selected a CloudWatch metric, then you haven’t yet specified which statistic you want to use, and you should do so immediately.

When syncing data from CloudWatch to SignalFx, a 60-second sampling period is used. See AWS CloudWatch documentation for more detailed information.

Amazon Web Services namespaces 🔗

SignalFx imports the namespaces for Amazon Web Services using the dimension namespace. For most services, the namespaces take the format of “AWS/<name_of_service>”, e.g. “AWS/EC2” or “AWS/ELB”. This distinction is important when you want to use metrics that have the same name across services, such as CPUUtilization, but only for a single service (say, EC2 and not ECS).

To control the amount of data being imported, you can specify which namespaces you want to import, as well as what data to import or exclude from each namespace. For more information, see Specifying Namespaces, Metrics, and Tags.

Uniquely identifying Amazon Web Services instances 🔗

The Amazon Web Services instance ID is not a unique identifier. To uniquely identify an AWS instance, concatenate instanceId, region, and accountID separated by underscores “_”, as follows:

instanceId_region_accountID

We’ve provided a script that you can use to print a unique ID for an Amazon Web Services instance, suitable for use in SignalFx. Find the script here.

To construct the identifier manually, first obtain metadata values for each instance from the following URL:

curl http://169.254.169.254/latest/dynamic/instance-identity/document

Here’s an example response from that URL:

{
"devpayProductCodes" : null,
 "privateIp" : "10.1.15.204",
 "availabilityZone" : "us-east-1a",
 "version" : "2010-08-31",
 "accountId" : "134183635603",
 "instanceId" : "i-a99f9802",
 "billingProducts" : null,
 "instanceType" : "c3.2xlarge",
 "pendingTime" : "2015-09-02T16:45:40Z",
 "imageId" : "ami-2ef44746",
 "kernelId" : null,
 "ramdiskId" : null,
 "architecture" : "x86_64",
 "region" : "us-east-1"
}

Parse the response to obtain values for Amazon Web Services instanceId, region, and accountId. Finally, concatenate them separated by underscores “_” as follows: instanceId_region_accountId. Use this identifier as the value for ”sfxdim_AWSUniqueId” in the procedure shown above.