CUDA memory bandwidth when reading a limited number of finite-sized chunks?

MWB 0
Viewed 53

Knowing hardware limits is useful for understanding if your code is performing optimally. The global device memory bandwidth limits how many bytes you can read per second, and you can approach this limit if the chunks you are reading are large enough.

But suppose you are reading, in parallel, N chunks of D bytes each, scattered in random locations in global device memory. Is there a useful formula limiting how much of the bandwidth you'd be able to achieve then?

cuda
2 Answers

let's assume:

  • we are talking about accesses from device code
  • a chunk of D bytes means D contiguous bytes
  • when reading a chunk, the read operation is fully coalesced - those bytes are read 4 bytes per thread, by however many adjacent threads in the block are predicted by D/4.
  • the temporal and spatial characteristics are such that no two chunks are within 32 bytes of each other - either they are all gapped by that much, or else the distribution of loads in time is such that the L2 doesn't provide any benefit. Pretty much saying the L2 hitrate is zero. This seems evident in your statement "global device memory bandwidth" - if the L2 hitrate is not zero, you're not measuring (purely) global device memory bandwidth
  • we are talking about a relatively recent GPU architecture, say Pascal or newer, or else for an older architecture the L1 is disabled for global loads. Pretty much saying the L1 hitrate is zero.
  • the overall footprint is not so large as to thrash the TLB
  • the starting address of each chunk is aligned to a 32-byte boundary (&)
  • your GPU is sufficiently saturated with warps and blocks to make full use of all resources (e.g. all SMs, all SM partitions, etc.)
  • the actual chunk access pattern (distribution of addresses) does not result in partition camping or some other hard-to-predict effect

In that case, you can simply round the chunk size D up to the next multiple of 32, and do a calculation based on that. What does that mean?

The predicted bandwidth (B) is:

Bd = the device memory bandwidth of your GPU as indicated by deviceQuery
B = Bd/(((D+31)/32)*32)

And the resultant units there is chunks/sec. (bytes/sec divided by bytes/chunk). The second division operation shown is "integer division", i.e. dropping any fractional part.

(&) In the case where we don't want this assumption, the worst case is to add an additional 32-byte segment per chunk. The formula then becomes:

B = Bd/((((D+31)/32)+1)*32)

note that this condition cannot apply when the chunk size is less than 34 bytes.

All I am really doing here is calculating the number of 32-byte DRAM transactions that would be generated by a stream of such requests, and using that to "derate" the observed peak (100% coalesced/100% utilized) case.

Robert Crovella0

Under @RobertCrovella's assumptions, and assuming the chunk sizes are multiples of 32 bytes and chunks are 32-byte aligned, you will get the same bandwidth as for a single chunk - as Robert's formula tells you. So, no benefit and no detriment.

But ensuring these assumptions hold is often not trivial (even merely ensuring coalesced memory reads).

einpoklum0
Related