x86 - Can the Intel performance monitor counters be used to measure memory bandwidth?

itemprop="text">

Can the Intel PMU be used
to measure per-core read/write memory bandwidth usage? Here "memory" means to DRAM
(i.e., not hitting in any cache level).

class="post-text" itemprop="text">

class="normal">Answer

Yes, this
is possible, although it is not necessarily as straightforward as programming the usual
PMU counters.

One approach is to use the
programmable memory controller counters which are accessed via PCI space. A good place
to start is by examining Intel's own implementation in
pcm-memory at href="https://github.com/opcm/pcm/blob/master/pcm-memory.cpp" rel="nofollow
noreferrer">pcm-memory.cpp. This app shows you the per-socket or
per-memory-controller throughput, which is suitable for some uses. In particular, the
bandwidth is shared among all cores, so on a quiet machine you can assume most of the
bandwidth is associated with the process under test, or if you wanted to monitor at the
socket level it's exactly what you want.

The
other alternative is to use careful programming of the "offcore repsonse" counters.
These, as far as I know, relate to traffic between the L2 (the last core-private cache)
and the rest of the system. You can filter by the result of the offcore response, so you
can use a combination of the various "L3 miss" events and multiply by the cache line
size to get a read and write bandwidth. The events are quite fine grained, so you can
further break it down by the what caused the access in the first place: instruction
fetch, data demand requests, prefetching, etc,
etc.

The offcore response counters generally lag
behind in support by tools like perf and
likwid but at least recent versions seem to have reasonable
support, even for client parts like SKL.