Docs » Integrations Guide » Monitor Kubernetes » Kubernetes Advanced Installation

Kubernetes Advanced Installation 🔗

The SignalFx Smart Agent was first written for Kubernetes and is relatively easy to set up in a cluster. The agent runs on each node and monitors services running on those same nodes to minimize cross-node traffic.

These instructions install the SignalFx Smart Agent on your Kubernetes cluster. The Agent is pre-configured to auto-discover SignalFx-supported integrations which it monitors.

For a simplified method of installing using helm, see the Smart Agent for Kubernetes Quick Installation.

Installation of Smart Agent for Kubernetes using kubectl 🔗

Installing the Smart Agent using kubectl involves three simple steps:

  1. Store a SignalFx access token in a Kubernetes secret.
  2. Configure the Kubernetes daemon set and the SignalFx Smart Agent for your cluster.
  3. Run a command on your Kubernetes cluster to install the Smart Agent and begin streaming data into SignalFx.

Installation details 🔗

Step 1. On the server that typically runs kubectl, enter the following command to create a Kubernetes secret (named signalfx-agent) containing your organization access token (named access-token):

$ kubectl create secret generic --from-literal access-token=MY_ACCESS_TOKEN signalfx-agent

Step 2. Download these configuration files from the SignalFx Github repository to the machine on which you usually run kubectl, and modify them as indicated under each filename below.

  • In this file: daemonset.YAML

    • For RBAC-enabled cluster, look up required permissions for Smart Agent and add them.

    • For Rancher If the Rancher nodes are behind a proxy, ensure that the Docker engine has the proxy configured so that it can pull the signalfx-agent Docker image from quay.io. See the Rancher v1.6 documentation or Rancher v2.x documentation for details on how to configure the proxy.

    • The cAdvisor monitor runs on port 9344 instead of the standard port 4194. Use the following configuration for the cadvisor monitor:

      monitors:
      - type: cadvisor
       cadvisorURL: http://localhost:9344
      
    • For OpenShift: If you cannot use the default namespace, modify each resource and then the cluster administrator can run these commands:

      oc create serviceaccount signalfx-agent
      oc adm policy add-cluster-role-to-user anyuid system:serviceaccount:default:signalfx-agent
      
      oc edit scc privileged
      users: ...
      - system:serviceaccount:default:signalfx-agent
      
      serviceAccountName: signalfx-agent
      
  • In this file: configmap.yaml

    • You need to assign a unique name to each Kubernetes cluster; choose a name and use a text editor to replace MY-CLUSTER with the name you choose.
    • By default, the Smart Agent will send data to the us0 realm. If you are not in this realm, you will need to explicitly set the signalFxRealm option in the agent configuration. To determine if you are in a different realm, check your profile page in the SignalFx web application.
    • If you want to stop docker and cadvisor metrics being sent from certain containers, use the datapointsToExclude option in the monitor config files. For more information, see filtering.
  • In this file: serviceaccount.yaml

    • No changes.
  • In this file: clusterrole.yaml

    • No changes.
  • In this file: clusterrolebinding.yaml

    • Change MY_AGENT_NAMESPACE or the namespace in the service account token reference to the namespace in which you are deploying the agent. For example: cloudwatch.

Step 3. On the Kubernetes cluster run:

cat *.yaml | kubectl apply -f-

Your installation is complete.

Verify Your Installation 🔗

Data will begin to stream into your SignalFx application. Click the Dashboards Infrastructure Navigator tab to see evidence of this data presented as the health of your nodes. If you don’t see data arriving, check the logs on an agent container to see if there are any errors. You can also execute the command signalfx-agent status as shown in the example command below in any of the Smart Agent pods to get a diagnostic output from the Smart Agent.

while read -r line; do kubectl exec --namespace `echo $line` signalfx-agent status; done <<< `kubectl get pods -l app=signalfx-agent --all-namespaces --no-headers | tr -s " " | cut -d " " -f 1,2`

Troubleshooting 🔗

Look at the logs (the last 20 lines of logs from all agent pods running in the cluster) using the example command below.

while read -r line; do echo "\n`echo $line | cut -d " " -f 2`:" ; kubectl logs --namespace `echo $line` --tail 20 ; done <<< `kubectl get pods -l app=signalfx-agent --all-namespaces --no-headers | tr -s " " | cut -d " " -f 1,2`

If you see errors like the ones below in the Smart Agent logs:

Couldn't get machine info: Kubelet request failed - "401 Unauthorized", response:"Unauthorized"

Couldn't get cAdvisor container stats" error="failed to get all container stats from KubeletURL "https://localhost:10250/stats/container/": Kubelet request failed - "401 Unauthorized", response: "Unauthorized"

This means that the Smart Agent cannot authenticate to the kubelet. Assuming you have the ClusterRole and ClusterRoleBinding properly applied to the Smart Agent container service account, this could indicate that the kubelet doesn’t honor RBAC authentication. Many times in this case, the kubelet will expose a separate endpoint on port 10255 that allows reading stats and metrics about the kubelet. You can configure the Smart Agent to read from this port by replacing the original kubelet stats monitor config in the configmap.yaml with the following:

monitors:
- type: kubelet-stats
  kubeletAPI:
    authType: none
    url: http://localhost:10255

To Stop Using Host Networking

The agent runs with host networking by default (the hostNetwork: true option in the DaemonSet). If you want to stop using host networking for some reason, for example, the agent does not reliably address service pods or you have DNS resolution issues, you can make the agent run with its own network namespace by doing the following in the daemonset.yaml file:

This requires version 3.6.2 or later of Smart Agent.

  1. Change hostNetwork: true to hostNetwork: false in the DaemonSet.
  2. Remove the dnsPolicy setting or change it to dnsPolicy: ClusterFirst.
  3. Add the item hostname: ${MY_NODE_NAME} under agent.yaml in the agent ConfigMap.

Configure the kubelet-stats monitor to use the node name as the hostname by using the following config:

- type: kubelet-stats
  kubeletAPI:
    url: https://${MY_NODE_NAME}:10250
    authType: serviceAccount
  1. If you have a non-standard kubelet-stats config, alter this accordingly. Note that this requires that node names are valid DNS hosts as well and it will not work if node names are not resolvable. Cluster firewalls also have to allow for traffic from the pod network to the kubelets.

Service Auto Discovery 🔗

The SignalFx agent that is able to monitor Kubernetes environments is pre-configured to include most of the integrations that SignalFx supports out of the box. Using customizable rules that are based on the container image name and service port, you can automatically start monitoring the microservices running in the containers. Each integration has a default configuration that you can customize for your environment by creating a new integration configuration file.

For more information, see Endpoint Discovery.

Master Nodes 🔗

The DaemonSet provided with Smart Agent includes a set of tolerations for master nodes that should work across multiple K8s versions. If your master node does not use the taints included in the provided daemonset, you should replace the tolerations with your cluster master taint so that the Smart Agent will run on the master node(s).

Release Information 🔗

See the values.yaml file for more information about how to configure releases.