Ten years ago, most computers were desktop computers designed for best performances and their CPU frequency was fixed. Nowadays, most devices are embedded and use low power consumption processors like ARM CPUs. The power consumption now matters more than performance peaks.
Intel CPUs evolved from a single core to multiple physical cores in the same package and got new features: Hyper-threading to run two threads on the same physical core and Turbo Boost to maximum performances. CPU cores can be completely turned off (CPU HALT, frequency of 0) temporarily to reduce the power consumption, and the frequency of cores changes regulary depending on many factors like the workload and temperature. The power consumption is now an important part in the design of modern CPUs.
Warning! This article is a summary of what I learnt last weeks from random articles. It may be full of mistakes, don't hesitate to report them, so I can enhance the article! It's hard to find simple articles explaining performances of modern Intel CPUs, so I tried to write mine.
Tools used in this article
This article mentions various tools. Commands to install them on Fedora 24:
dnf install -y util-linux:
dnf install -y kernel-tools:
sudo dnf install -y msr-tools:
Other interesting tools, not used in this article: i7z (sadly no more maintained), lshw, dmidecode, sensors.
The sensors tool is supposed to report the current CPU voltage, but it doesn't provide this information on my computers. At least, it gives the temperature of different components, but also the speed of fans.
Example of Intel CPUs
My laptop CPU: /proc/cpuinfo
On Linux, the most common way to retrieve information on the CPU is to read /proc/cpuinfo. Example on my laptop:
selma$ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel model name : Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz cpu MHz : 1200.214 ... processor : 1 vendor_id : GenuineIntel model name : Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz cpu MHz : 3299.882 ...
"i7-3520M" CPU is a model designed for Mobile Platforms (see the "M" suffix). It was built in 2012 and is the third generation of the Intel i7 microarchitecture: Ivy Bridge.
The CPU has two physical cores, I disabled HyperThreading in the BIOS.
The first strange thing is that the CPU announces "2.90 GHz" but Linux reports 1.2 GHz on the first core, and 3.3 GHz on the second core. 3.3 GHz is greater than 2.9 GHz!
My desktop CPU: CPU topology with lscpu
smithers$ cat /proc/cpuinfo processor : 0 physical id : 0 core id : 0 ... model name : Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz cpu cores : 4 ... processor : 1 physical id : 0 core id : 1 ... (...) processor : 7 physical id : 0 core id : 3 ...
The CPU i7-2600 is the 2nd generation: Sandy Bridge microarchitecture. There are 8 logical cores and 4 physical cores (so with Hyper-threading).
The lscpu renders a short table which helps to understand the CPU topology:
smithers$ lscpu -a -e CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ 0 0 0 0 0:0:0:0 yes 3800.0000 1600.0000 1 0 0 1 1:1:1:0 yes 3800.0000 1600.0000 2 0 0 2 2:2:2:0 yes 3800.0000 1600.0000 3 0 0 3 3:3:3:0 yes 3800.0000 1600.0000 4 0 0 0 0:0:0:0 yes 3800.0000 1600.0000 5 0 0 1 1:1:1:0 yes 3800.0000 1600.0000 6 0 0 2 2:2:2:0 yes 3800.0000 1600.0000 7 0 0 3 3:3:3:0 yes 3800.0000 1600.0000
There are 8 logical CPUs (CPU 0..7), all on the same node (NODE 0) and the same socket (SOCKET 0). There are only 4 physical cores (CORE 0..3). For example, the physical core 2 is made of the two logical CPUs: 2 and 6.
Using the L1d:L1i:L2:L3 column, we can see that each pair of two logical cores share the same physical core caches for levels 1 (L1 data, L1 instruction) and 2 (L2). All physical cores share the same cache level 3 (L3).
A new CPU driver intel_pstate was added to the Linux kernel 3.9 (April 2009). First, it only supported SandyBridge CPUs (2nd generation), Linux 3.10 extended it to Ivybridge generation CPUs (3rd gen), and so on and so forth.
This driver supports recent features and thermal control of modern Intel CPUs. Its name comes from P-states.
The processor P-state is the capability of running the processor at different voltage and/or frequency levels. Generally, P0 is the highest state resulting in maximum performance, while P1, P2, and so on, will save power but at some penalty to CPU performance.
It is possible to force the legacy CPU driver (acpi_cpufreq) using intel_pstate=disable option in the kernel command line.
- Documentation of the intel-pstate driver
- Some basics on CPU P states on Intel processors (2013) by Arjan van de Ven (Intel)
- Balancing Power and Performance in the Linux Kernel talk at LinuxCon Europe 2015 by Kristen Accardi (Intel)
- What exactly is a P-state? (Pt. 1) (2008) by Taylor K. (Intel)
Idle states: C-states
C-states are idle power saving states, in contrast to P-states, which are execution power saving states.
During a P-state, the processor is still executing instructions, whereas during a C-state (other than C0), the processor is idle, meaning that nothing is executing.
- C0 is the operational state, meaning that the CPU is doing useful work
- C1 is the first idle state
- C2 is the second idle state: The external I/O Controller Hub blocks interrupts to the processor.
When a logical processor is idle (C-state except of C0), its frequency is typically 0 (HALT).
The cpupower idle-info command lists supported C-states:
selma$ cpupower idle-info CPUidle driver: intel_idle CPUidle governor: menu analyzing CPU 0: Number of idle states: 6 Available idle states: POLL C1-IVB C1E-IVB C3-IVB C6-IVB C7-IVB ...
The cpupower monitor shows statistics on C-states:
smithers$ sudo cpupower monitor -m Idle_Stats |Idle_Stats CPU | POLL | C1-S | C1E- | C3-S | C6-S 0| 0,00| 0,19| 0,09| 0,58| 96,23 4| 0,00| 0,00| 0,00| 0,00| 99,90 1| 0,00| 2,34| 0,00| 0,00| 97,63 5| 0,00| 0,00| 0,17| 0,00| 98,02 2| 0,00| 0,00| 0,00| 0,00| 0,00 6| 0,00| 0,00| 0,00| 0,00| 0,00 3| 0,00| 0,00| 0,00| 0,00| 0,00 7| 0,00| 0,00| 0,00| 0,00| 49,97
In 2005, Intel introduced SpeedStep, a serie of dynamic frequency scaling technologies to reduce the power consumption of laptop CPUs. Turbo Boost is an enhancement of these technologies, now also used on desktop and server CPUs.
Turbo Boost allows to run one or many CPU cores to higher P-states than usual. The maximum P-state is constrained by the following factors:
- The number of active cores (in C0 or C1 state)
- The estimated current consumption of the processor (Imax)
- The estimated power consumption (TDP - Thermal Design Power) of processor
- The temperature of the processor
Example on my laptop:
selma$ cat /proc/cpuinfo model name : Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz ... selma$ sudo cpupower frequency-info analyzing CPU 0: driver: intel_pstate ... boost state support: Supported: yes Active: yes 3400 MHz max turbo 4 active cores 3400 MHz max turbo 3 active cores 3400 MHz max turbo 2 active cores 3600 MHz max turbo 1 active cores
The CPU base frequency is 2.9 GHz. If more than one physical cores is "active" (busy), their frequency can be increased up to 3.4 GHz. If only 1 physical core is active, its frequency can be increased up to 3.6 GHz.
In this example, Turbo Boost is supported and active.
See also the Linux cpu-freq documentation on CPU boost.
Turbo Boost MSR
The bit 38 of the Model-specific register (MSR) 0x1a0 can be used to check if the Turbo Boost is enabled:
selma$ sudo rdmsr -f 38:38 0x1a0 0
0 means that Turbo Boost is enabled, whereas 1 means disabled (no turbo). (The -f 38:38 option asks to only display the bit 38.)
If the command doesn't work, you may have to load the msr kernel module:
sudo modprobe msr
Note: I'm not sure that all Intel CPU uses the same MSR.
Turbo Boost can also be disabled at runtime in the intel_pstate driver.
Check if Turbo Boost is enabled:
selma$ cat /sys/devices/system/cpu/intel_pstate/no_turbo 0
where 0 means that Turbo Boost is enabled. Disable Turbo Boost:
selma$ echo 1|sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
CPU flag "ida"
It looks like the Turbo Boost status (supported or not) can also be read by the CPUID(6): "Thermal/Power Management". It gives access to the flag Intel Dynamic Acceleration (IDA).
The ida flag can also be seen in CPU flags of /proc/cpuinfo.
Read the CPU frequency
General information using cpupower frequency-info:
selma$ cpupower -c 0 frequency-info analyzing CPU 0: driver: intel_pstate ... hardware limits: 1.20 GHz - 3.60 GHz ...
The frequency of CPUs is between 1.2 GHz and 3.6 GHz (the base frequency is 2.9 GHz on this CPU).
Get the frequency of CPUs: turbostat
It looks like the most reliable way to get a relialistic estimation of the CPUs frequency is to use the tool turbostat:
selma$ sudo turbostat CPU Avg_MHz Busy% Bzy_MHz TSC_MHz - 224 7.80 2878 2893 0 448 15.59 2878 2893 1 0 0.01 2762 2893 CPU Avg_MHz Busy% Bzy_MHz TSC_MHz - 139 5.65 2469 2893 0 278 11.29 2469 2893 1 0 0.01 2686 2893 ...
- Avg_MHz: average frequency, based on APERF
- Busy%: CPU usage in percent
- Bzy_MHz: busy frequency, based on MPERF
- TSC_MHz: fixed frequency, TSC stands for Time Stamp Counter
APERF (average) and MPERF (maximum) are MSR registers that can provide feedback on current CPU frequency.
Other tools to get the CPU frequency
It looks like the following tools are less reliable to estimate the CPU frequency.
selma$ grep MHz /proc/cpuinfo cpu MHz : 1372.289 cpu MHz : 3401.042
In April 2016, Len Brown proposed a patch modifying cpuinfo to use APERF and MPERF MSR to estimate the CPU frequency: x86: Calculate MHz using APERF/MPERF for cpuinfo and scaling_cur_freq.
The tsc clock source logs the CPU frequency in kernel logs:
selma$ dmesg|grep 'MHz processor' [ 0.000000] tsc: Detected 2893.331 MHz processor
selma$ for core in $(seq 0 1); do sudo cpupower -c $core frequency-info|grep 'current CPU'; done current CPU frequency: 3.48 GHz (asserted by call to hardware) current CPU frequency: 3.40 GHz (asserted by call to hardware)
selma$ sudo cpupower monitor -m 'Mperf' |Mperf CPU | C0 | Cx | Freq 0| 4.77| 95.23| 1924 1| 0.01| 99.99| 1751
Modern Intel CPUs use various technologies to provide best performances without killing the power consumption. It became harder to monitor and understand CPU performances, than with older CPUs, since the performance now depends on much more factors.
It also becomes common to get an integrated graphics processor (IGP) in the same package, which makes the exact performance even more complex to predict, since the IGP produces heat and so has an impact on the CPU P-state.
I should also explain that P-state are "voted" between CPU cores, but I didn't understand this part. I'm not sure that understanding the exact algorithm matters much. I tried to not give too much information.
Annex: AMT and the ME (power management coprocessor)
Computers with Intel vPro technology includes Intel Active Management Technology (AMT): "hardware and firmware technology for remote out-of-band management of personal computers". AMT has many features which includes power management.
Management Engine (ME) is the hardware part: an isolated and protected coprocessor, embedded as a non-optional part in all current (as of 2015) Intel chipsets. The coprocessor is a special 32-bit ARC microprocessor (RISC architecture) that's physically located inside the PCH chipset (or MCH on older chipsets). The coprocessor can for example be found on Intel MCH chipsets Q35 and Q45.
See Intel x86s hide another CPU that can take over your machine (you can't audit it) for more information on the coprocessor.
More recently, the Intel Xeon Phi CPU (2016) also includes a coprocessor for power management. I didn't understand if it is the same coprocessor or not.