Perf shows L1-dcache-load-misses in a block with no memory access

user_8276973 0
Viewed 516

Below is a block of code that perf record flags as responsible for 10% of all L1-dcache misses, but the block is entirely movement between zmm registers. This is the perf command string:

perf record -e L1-dcache-load-misses -c 10000 -a -- ./Program_to_Test.exe

The code block:

Round:
vmulpd zmm1,zmm0,zmm28
VCVTTPD2QQ zmm0{k7},zmm1
VCVTUQQ2PD zmm2{k7},zmm0
vsubpd zmm3,zmm1,zmm2
vmulpd zmm4,zmm3,zmm27
VCVTTPD2QQ zmm5{k7}{z},zmm4

VPCMPGTQ k2,zmm5,zmm26
VPCMPEQQ k3 {k7},zmm5,zmm26
KADDQ k1,k2,k3

VCVTQQ2PD zmm2{k7},zmm0
VDIVPD zmm1{k7},zmm2,zmm28 ; Divide by 100
VPXORQ zmm2{k7},zmm2,zmm2
vmovupd zmm2,zmm1
VADDPD zmm2{k1},zmm1,zmm25

I get similar results for that code block with other L1 measures such as l1d.replacement.

My question is, how can a block that is only zmm register movement generate L1 cache misses? I didn't think registers go to memory at all. In fact, the last memory access is 10 instructions above this block of code; the other 9 instructions are all register-to-register instructions.

perf x86-64 profiling linux performance
1 Answers

The event L1-dcache-load-misses is mapped to L1D.REPLACEMENT on Sandy Bridge and later microarchitectures (or mapped to a similar event on older microarchitectures). This event doesn't support precise sampling, which means that a sample can point to an instruction that couldn't have generated the event being sampled on. (Note that L1-dcache-load-misses is not supported on any current Atom.)

Starting with Linux 3.11 running on a Haswell+ or Silvermont+ microarchitecture, samples can be captured with eventing instruction pointers by specifying a sampling event that meets the following two conditions:

  • The event supports precise sampling. You can use, for example, any of the events that represent memory uop or instruction retirement. The exact names and meaning of the events depends on the microarchtiecture. Refer to the Intel SDM Volume 3 for more information. There is no event that supports precise sampling and has the same exact meaning as L1D.REPLACEMENT. On processors that support Extended PEBS, only a subset of PEBS events support precise sampling.
  • The precise sampling level is enabled on the event. In Linux perf, this can be done by appending ":pp" to the event name or raw event encoding, or "pp" after the terminating slash of a raw event specified in the PMU syntax. For example, on Haswell, the event mem_load_uops_retired.l1_miss:pp can be specified to Linux perf.

With such an event, when the event counter overflows, the PEBS hardware is armed, which means that it's now looking for the earliest possible opportunity to collect a precise sample. When there is at least one instruction that will cause an event during this window of time, the PEBS hardware will be eventually triggered by one of these instructions with bias toward high-latency instructions. When the instruction that triggeres PEBS retires, the PEBS microcode routine will execute and captures a PEBS record, which contains among other things the IP of the instruction that triggered PEBS (which is different from the architectural IP). The instruction pointer (IP) used by perf to display the results is this eventing IP. (I noticed there can be a negligible number of samples pointing to instructions that couldn't have caused the event.)

On older mircroarchitecures (before Haswell and Silvermont), the "pp" precise sampling level is also supported. PEBS on these processors will only capture the architectural event, which points to the static instruction that immediately follows the PEBS triggering instruction in program order. Linux perf uses LBR, if possible, which contains source-target IP pairs to determine if that captured IP is a target of a jump. If that was the case, it will add the source IP as the eventing IP to the sample record.

Some microarchitectures support one or more events with better sampling distribution (how much better depends on the microarchitecture, the event, the counter, and the instructions being executed at the time in which the counter is about to overflow). In Linux perf, precise distribution can be enabled, if supported, by specifying the precise level "ppp."

user_42306180
Related