Docs » Integrations Guide » Monitor Kubernetes » Kubernetes Advanced Installation

Kubernetes Advanced Installation πŸ”—

The SignalFx Smart Agent was first written for Kubernetes and is relatively easy to set up in a cluster. The Smart Agent runs on each node and monitors services running on those same nodes to minimize cross-node traffic.

These instructions install the SignalFx Smart Agent on your Kubernetes cluster using kubectl. The Smart Agent is pre-configured to auto-discover SignalFx-supported integrations that it monitors.

For a simplified method of installing using Helm, see Kubernetes Quick Installation.

Smart Agent Install for Kubernetes using kubectl πŸ”—

This installation involves three simple steps:

  1. Store a SignalFx access token in a Kubernetes secret.
  2. Configure the Kubernetes daemon set and the SignalFx Smart Agent for your cluster.
  3. Run a command on your Kubernetes cluster to install the Smart Agent and begin streaming data into SignalFx.

Step 1. On the server that typically runs kubectl, run this command to create a Kubernetes secret (named signalfx-agent) containing your organization access token (named access-token):

$ kubectl create secret generic --from-literal access-token=MY_ACCESS_TOKEN signalfx-agent

Step 2. Download these configuration files from the SignalFx Github repository to the machine on which you usually run kubectl, and modify them as indicated under each filename below.

  • In daemonset.YAML:

    • For RBAC-enabled cluster, look up required permissions for Smart Agent and add them.

    • For Rancher, if the Rancher nodes are behind a proxy, ensure that the Docker engine has the proxy configured so that it can pull the signalfx-agent Docker image from See the Rancher v1.6 documentation or Rancher v2.x documentation for details on how to configure the proxy.

    • The cAdvisor monitor runs on port 9344 instead of the standard port 4194. Use the following configuration for the cadvisor monitor:

      - type: cadvisor
       cadvisorURL: http://localhost:9344
    • For OpenShift, if you cannot use the default namespace, modify each resource, and then the cluster administrator can run these commands:

      oc create serviceaccount signalfx-agent
      oc adm policy add-cluster-role-to-user anyuid system:serviceaccount:default:signalfx-agent
      oc edit scc privileged
      users: ...
      - system:serviceaccount:default:signalfx-agent
      serviceAccountName: signalfx-agent
  • In configmap.yaml:

    • You need to assign a unique name to each Kubernetes cluster; choose a name and use a text editor to replace MY-CLUSTER with the name you choose.
    • By default, the Smart Agent will send data to the us0 realm. If you are not in this realm, you will need to explicitly set the signalFxRealm option in the agent configuration. To determine if you are in a different realm, check your profile page in the SignalFx web application.
    • If you want to stop docker and cadvisor metrics being sent from certain containers, use the datapointsToExclude option in the monitor config files. For more information, see filtering.
  • In serviceaccount.yaml:

    • No changes.
  • In clusterrole.yaml:

    • No changes.
  • In clusterrolebinding.yaml, configmap-role.yaml, and configmap-rolebinding.yaml:

    • Change MY_AGENT_NAMESPACE or the service account token reference to the namespace in which you are deploying the agent. For example, monitoring.

Step 3. Run this command on your Kubernetes cluster.

cat *.yaml | kubectl apply -f-

Your installation is complete.

Verify Your Installation πŸ”—

Data will begin to stream into your SignalFx application. Click Dashboards in the navigation bar to see evidence of this data presented as the health of your nodes. If you don’t see data arriving, check the logs on an agent container to see if there are any errors preventing data from streaming to the SignalFx web application.

Run the signalfx-agent status command inside any of the Smart Agent containers to get a diagnostic output from the Smart Agent to quickly see what services the Smart Agent has discovered.

while read -r line; do kubectl exec --namespace `echo $line` signalfx-agent status; done <<< `kubectl get pods -l app=signalfx-agent --all-namespaces --no-headers | tr -s " " | cut -d " " -f 1,2`

Configuration πŸ”—

The SignalFx Smart Agent for Kubernetes comes pre-packaged with a set of monitors that it uses to collect metrics. Each monitor is pre-configured to collect a default set of metrics. For a list of Helm chart configuration options, see values.yaml. For instructions on how to configure monitors to collect custom metrics, see the detailed descriptions for each monitor.

Troubleshooting πŸ”—

Run this command to view the last 20 lines of Smart Agent logs from all agent pods running in the cluster.

while read -r line; do echo "\n`echo $line | cut -d " " -f 2`:" ; kubectl logs --namespace `echo $line` --tail 20 ; done <<< `kubectl get pods -l app=signalfx-agent --all-namespaces --no-headers | tr -s " " | cut -d " " -f 1,2`

If you see errors like the following, the Smart Agent cannot authenticate to the kubelet.

Couldn't get machine info: Kubelet request failed - "401 Unauthorized", response:"Unauthorized"

"Couldn't get machine info: ... Get https://localhost:10255/spec/... :10255: connect: connection refused"

Couldn't get cAdvisor container stats" error="failed to get all container stats from Kubelet URL "https://localhost:10250/stats/container/": Kubelet request failed - "401 Unauthorized", response: "Unauthorized"

Couldn't get cAdvisor container stats" error="failed to get all container stats from Kubelet URL "https://localhost:10255/stats/container/": "10255: connect: connection refused"

If you have ClusterRole and ClusterRoleBinding properly applied to the Smart Agent container service account, this could indicate that the kubelet doesn’t honor RBAC authentication. Often in this case, the kubelet will expose a separate endpoint on either port 10250 or 10255 that allows reading stats and metrics about the kubelet.

You can configure the Smart Agent to read from the open port (either 10250 or 10255) by replacing the original kubelet stats monitor config in configmap.yaml with the following:

- type: kubelet-stats
    authType: none
    url: http://localhost:10255

To Stop Using Host Networking πŸ”—

The Smart Agent runs with host networking by default (the hostNetwork: true option in the DaemonSet). For infrastructure correlation with Splunk APM, you have to enable the Smart Agent with host networking. Infrastructure correlation allows you to get a bird’s eye view of how your microservices interact with and depend on each other and isolate Kubernetes resources when analyzing service degredation.

If you want to stop using host networking for some reason, for example, the agent does not reliably address service pods or you have DNS resolution issues, you can make the Smart Agent run with its own network namespace by doing the following in the daemonset.yaml file:

This requires version 3.6.2 or later of Smart Agent.

  1. Change hostNetwork: true to hostNetwork: false in the DaemonSet.
  2. Remove the dnsPolicy setting or change it to dnsPolicy: ClusterFirst.
  3. Add the item hostname: ${MY_NODE_NAME} under agent.yaml in the agent ConfigMap.

Configure the kubelet-stats monitor to use the node name as the hostname by using the following config:

- type: kubelet-stats
    url: https://${MY_NODE_NAME}:10250
    authType: serviceAccount
  1. If you have a non-standard kubelet-stats config, alter this accordingly. This requires that node names are valid DNS hosts as well and it will not work if node names are not resolvable. Cluster firewalls also have to allow for traffic from the pod network to the kubelets.

Service Auto Discovery πŸ”—

The Smart Agent that is able to monitor Kubernetes environments is pre-configured to include most of the integrations that SignalFx supports out of the box. Using customizable rules that are based on the container image name and service port, you can automatically start monitoring the microservices running in the containers. Each integration has a default configuration that you can customize for your environment by creating a new integration configuration file.

For more information, see Endpoint Discovery.

Master Nodes πŸ”—

The DaemonSet provided with Smart Agent includes a set of tolerations for master nodes that should work across multiple Kubernetes versions. If your master node does not use the taints included in the provided daemonset, you should replace the tolerations with your cluster master taint so that the Smart Agent will run on the master node(s).