Why We Should Get Rid of Average CPU Utilization

A post came up on Hacker News (thread) discussing that CPU utilisation should be removed, looking at the process of troubleshooting an application timeout issue that was plaguing their service. While many of the points of the article are valid, I think the wider point is that CPU utilisation should not be the primary metric used to identify issues or bottlenecks in an environment.

What is CPU Utilisation?

First of all, I think it’s worth looking at what CPU utilisation is from a metrics perspective. Utilisation for CPU’s is what I would call an aggregated metric, it’s not a specific state that a CPU is in, but a collection of all non-idle states as a single summed value. Specifically for Linux, there are 10 different states a processor can be in, 9 of which are aggregated into “utilisation”. Where this becomes a problem is that a 40% utilisation that is mostly user time is significantly different to a 40% utilisation that is mostly iowait. Same total value, but very different experiences for the user. For Windows, you can actually exceed 100% CPU utilisation due to nuances of how the system calculates utilisation and the impacts of processor technologies allowing variable processor speeds depending on the workload. VMware’s vSphere has 23 CPU metrics across individual CPU’s and aggregate CPU’s in VM’s (along with 30 different metrics for memory). It also runs into the same problem as Windows where CPU usage is measured relative to the base processor frequency, allowing for over 100% utilisation for a given CPU. All of this to say that CPU utilisation is at best an imperfect measure for a system’s health, and at worst an unreliable measure as a primary diagnosis tool. The grounding of CPU utilisation as a primary metric is grounded in the pre-virtualisation days when operating systems were deployed 1:1 with their related hardware, applications were mostly single-threaded and we had certain assumptions on performance and underlying infrastructure that are no longer true today.

Is There a Better Measure?

If you’re looking for a single metric that will capture and distill everything down, no, there isn’t. In most cases when the CPU is being investigated you will need to look at a combination of all metrics made available to you and tweak accordingly. If you’re looking to create an alert to indicate a problem, your best option would be anomaly alerting across user, nice, system, iowait and possibly stolen - especially in shared virtualisation environments or when using burstable cloud instances. Spikes or drops across these metrics can be possible causes of application issues impacting users. If you were looking for static thresholds to measure against, checking for high iowait or stolen values are the ones to focus on, as high or low values for other metrics will vary depending on user activity across the environment and therefore are less appropriate to have a single good-or-bad metric for, as a high or low user % might just be due to it being a weekend and so on.

For Windows servers, potentially windows_cpu_idle_break_events_total, windows_cpu_processor_utility_total and windows_cpu_interrupts_total would provide a reasonable measure, as it will allow visibility of times woken from idle, which incur performance delays if happening regularly due to the time needed for the core to awaken, total time spent utilised by the CPU (despite the abovementioned issues with these numbers) and the count of CPU interrupts, which will highlight any excessive context switches at the OS layer that might indicate contention from the underlying applications. windows_cpu_time_total is also likely to be a good metric to measure against. As with the Linux component mentioned above, these are most likely best viewed as anomalies compared to a baseline, and any static thresholds to measure against are likely going to be a limited subset. My current thinking on this is measuring CPU interrupt count and CPU interrupt time as these are signs of internal application contention that will impact users and should ideally hold constant over time.