Docs » Getting Started » Concepts » SignalFlow Analytics Language

SignalFlow Analytics Language 🔗

The heart of the SignalFx platform is a streaming, real-time analytics engine that executes computations written in a flexible language named SignalFlow. Examples of the computations that are commonly used include:

  • Comparisons with historical norms, e.g. on a week-over-week basis
  • Population overviews using a distributed percentile chart
  • Detecting if the rate of change (or other metric expressed as a ratio, such as a service level objective) has exceeded a critical threshold
  • Finding correlated dimensions, e.g. to determine which service is most correlated with alerts for low disk space

For the most part, these computations are created using a simple Chart Builder UI that allows you to easily specify what metric time series you want to use and what analytical functions you want to apply to them. More advanced users can create SignalFlow programs directly via the SignalFx API.

SignalFlow includes a large library of built-in analytical functions that take a metrics time series as an input, perform computations on its datapoints, and output time series that are the result of the computation.

Aggregations and transformations 🔗

Many of the built-in analytical functions are able to perform computations on time series in charts and detectors in two ways: aggregations and transformations.

  • Aggregations operate across all of the datapoints at a single instance in time, e.g. the average (or mean) CPU utilization across a group of 5 servers at time t, t+1, t+2 etc. The output of an aggregation is a single MTS (metric time series) where each datapoint represents the aggregation of all the datapoints over a specific period of time.

    An additional option, Group By, is available for aggregations. If a group‑by is specified, MTS sharing values for properties named in the group-by criterion are aggregated together. For example, you can compute the average CPU load grouped by AWS instance type; specify the Mean function as an aggregation, and set AWS instance type as the group‑criterion. The output will show 1 MTS per AWS instance type.

  • Transformations operate in parallel on each MTS over a window of time and yield one output time series for each input time series. For example, the average CPU utilization for 5 servers over a moving window of one day will display five MTS; each output value will be the moving average for that MTS over the previous 24 hours.

    The two types of transformations available, moving window and calendar window, are discussed below. For various examples of how to use transformation analytics in charts, see Gaining Insight Through Analytics.

Moving window transformations 🔗

In the following chart, each line represents the Mean CPU utilization across 4 servers. The grey line (plot A) represents the mean value for each datapoint over the preceding minute (because the chart resolution is 1 minute, as shown next to the chart title). The magenta lines (plot B) represent the mean value for each datapoint over the preceding hour (moving window), because 1 hour is specified in the analytics function.

../../_images/aggr-transform-moving.png

Calendar window transformations 🔗

The Sum, Mean, Maximum, and Minimum analytics functions let you specify a calendar window for a transformation, instead of a moving window. In the following chart the magenta line (left Y-axis) shows the sum of all transactions over a moving window of 1 week (7 days). The green line (right Y-axis) shows the sum of the transactions over a calendar week, including partial values calculated throughout the week. As you might expect, the latter values increase over the course of a week, then reset at the beginning of the following week.

../../_images/moving-and-cal.png

When you add an analytics function with a calendar window to a plot, and the current time window is narrower than the cycle length that you specified on the function, SignalFx automatically optimizes the chart default time so a time range of at least one cycle can be displayed. Also, any dashboard override for time range is removed. A message will be displayed to inform you of this optimization; if you don’t accept the optimization, you may need to modify the time range manually to see the data you expect.

In order for a chart to show a value at the end of every calendar cycle, the cycle length must be a multiple of the chart resolution (shown to the right of the chart’s title). Some cycle lengths are fixed; a week is always 7 days. Others are variable; a month can be 28, 29, 30, or 31 days. SignalFx always tries to optimize the resolution such that the cycle length is a multiple of the chart resolution.

However, for some combinations of time range and chart display resolution (the density selector at the right side of the override bar), it may not be possible to use a resolution that guarantees a chart will show values perfectly aligned with cycle boundaries. For example, if a resolution of 1 day would result in more datapoints than can be shown on a chart, a resolution of two days may have to be used. This means that plotted values cannot line up with the end of a month that has 29 or 31 days, because neither value is a multiple of the 2‑day resolution. Such a situation is visually indicated by the resolution pill on a chart turning orange and showing a message in the on-hover tooltip. You can resolve this issue by changing the display resolution and/or viewing a narrower time range in the chart.

When using calendar time windows with transformations, the chart cannot have a resolution finer than 1 hour.

When specifying a calendar time window, you have a few options you can set. These are explained below.

../../_images/cal-window-options.png

Cycle length and start 🔗

These options are self-explanatory. Cycle length options include hour, day, week, month, and quarter.

For most cycle length options, you can specify a starting point. For example, for a cycle length of a quarter, you could specify that the first quarter starts in February instead of the default of January. The one exception is an hourly cycle length; hourly cycles always start at the top of the hour (minute 0).

Calendar time zone 🔗

For calendar windows, you must specify a calendar time zone. All calendar window functions in a chart use the same calendar time zone.

The time zone that you specify here is a per-chart (or per-detector) option that is independent of the visualization timezone that is set in your user profile; the time zone you set for a calendar window is used to determine the exact beginning and end of your chosen calendar window cycles.

For example, January in America/Los Angeles starts at a different instant relative to January in Asia/Tokyo. If SignalFx receives a datapoint with a timestamp near midnight UTC time on December 31, the calendar time zone determines whether that datapoint should count towards the calculation for December (which it would for Los Angeles) or the calculation for January (which it would for Tokyo).

The first time a calendar window function is specified on any plot in a chart, the visualization timezone from your profile is suggested as the value to use for the calendar timezone. However, you can select any time zone you need. The value set here can be viewed and changed in any calendar window function in the chart, as well as in the chart options tab.

Hide partial values 🔗

This setting lets you optimize the output of a calendar window function, based on whether you are interested only in the final values calculated at the ends of cycles (such as the sum of requests served every day) or final as well as partial values calculated during a cycle (such as the sum of requests served so far today). For example, if you have a cycle length of 1 day, hiding partial values means that you will only see one value for each day; you won’t see how values change during the course of the day.

Note that deselecting this option has no effect when cycle length is 1 hour, because a chart using calendar windows cannot have a resolution finer than 1 hour.

In the following chart, hiding partial values (the magenta bars) provides a better overview of how values actually compare on a day-to-day basis. Not hiding partial values (the green lines) shows how the mean changes during the course of each day.

The value you see at the start of each cycle represents the final value for the previous cycle. The magenta column at 12:00am February 15 (on the far left) represents the mean of the values seen over February 14, the column at 12:00am February 16 represents the mean of the values for February 15, and so on.

../../_images/cal-window-show-hide-2.png

Tip

Single value charts can be useful for visualizing calculations such as the maximum latency reported in the current day so far. To properly display these numbers, deselect “Hide partial values”.

Timeshift 🔗

If you select timeshift (available only when partial values are hidden) then the value from the end of a previous cycle will be output at the end of every calendar cycle. For example, if your cycle length is Month and you timeshift by one cycle, the datapoint at April 30 will represent the value from March 31, the datapoint at May 31 will represent the value from April 30, and so on.

Note that this timeshift option is aware of cycles such as month having variable lengths (March has more days than Feb) and will shift correctly to the end of a previous interval. By contrast, the standalone timeshift analytics function (see Using the Timeshift function to understand trends) always performs a fixed width shift, such as 30 days. For this reason, we recommend using the timeshift option available within a calendar window transformation when you want to shift to the ends of previous monthly or quarterly cycles.

A typical use case for using timeshift is to create a column chart that includes a plot with a calendar transformation that is not timeshifted, then clone that plot and add the timeshift option. This allows you to compare, say, the average value seen for a metric over the current week with the average seen over the previous week. In the following chart, plot A (the magenta columns on the left) shows sums over a day. Plot B (the green columns on the right) shows what the comparable values were 7 days ago.

../../_images/cal-window-timeshift.png

Other functions 🔗

Beyond functions that provide aggregations and transformations, SignalFx also offers a variety of other useful functions, such as Count, which counts the number of MTS that have values; Top and Bottom, which show the highest or lowest N number of values; and Exclude, which provides the ability to filter time series by value, rather than by source.

As with other analytical functions, these functions can be used in concert with others to produce more sophisticated computations. For example, Exclude can be used with Sum to achieve a result akin to the sumif() function found in popular spreadsheet applications.

For a detailed explanation of each function, see Analytics Reference Guide.

Expressions 🔗

On top of analytics functions, SignalFx also offers the ability to create arbitrary mathematical expressions that refer to preceding computations as variables, via either the Chart Builder UI or the SignalFlow API. These expressions can be used to define anything from ratios (e.g. ratio of HTTP response codes received that are 200s to those that are 400s or 500s) to derived metrics (e.g. cache.total = cache.hits + cache.misses), rates of change, or essentially anything that can be formulated by the user as an equation.