What is the L2 cache accessPolicyWindow introduced in CUDA 11

user_4794308 0
Viewed 233

New runtime API is introduced in CUDA 11 to fine-tune L2 access policy.

But I do not fully understand the meaning of policy like hitRatio and how they can be used in practice.

In particular, I saw in CUDA API Documentation that cudaAccessPolicyWindow:

Specifies an access policy for a window, a contiguous extent of memory beginning at base_ptr and ending at base_ptr + num_bytes. Partition into many segments and assign segments such that. sum of "hit segments" / window == approx. ratio. sum of "miss segments" / window == approx 1-ratio. Segments and ratio specifications are fitted to the capabilities of the architecture. Accesses in a hit segment apply the hitProp access policy. Accesses in a miss segment apply the missProp access policy.

My question:

  1. How is the contiguous extent of memory "partitioned" into segments? Are these partitions decided statically based on the hit prop such that the hit or miss attribute of segments will remain unchanged once they are assigned or there are some running counters which dynamically adjust the assignment e.g. on each access basis?

  2. How should these attributes be applied to optimize performance in practice? E.g. to put it in a over-simplified & naive way, suppose I have 1 MB L2 cache set aside, should I create a window of 1 MB for my most frequently used data and set hitRatio as 1 or should I create a window of 2 MB and set hitRatio as 0.5? Why?

cuda parallel-processing gpu
1 Answers

How is the contiguous extent of memory "partitioned" into segments? Are these partitions decided statically based on the hit prop such that the hit or miss attribute of segments will remain unchanged once they are assigned or there are some running counters which dynamically adjust the assignment e.g. on each access basis?

I don't think any of this is explicitly specified. However we can make some pretty solid (I believe) conjecture based on two ideas:

  1. NVIDIA GPU L2 caches for a "long time" have had a 32-byte basic line size. This is designed to align with the design of the DRAM subsystem which has 32-byte segment boundaries.

  2. The documentation states that which lines are chosen to be persistent in the cache as a "random" selection. This pretty much implies it is not tied to access patterns.

Assembling these ideas, then I would say that the partitioning is done at a granularity of no less then a L2 line (32 bytes) and it may be a higher granularity if needed by the L2 tag/TLB system. Most of these L2 details are generally unpublished by NVIDIA. Once selected at random, I would not expect the selection to change based on access pattern.

How should these attributes be applied to optimize performance in practice? E.g. to put it in a over-simplified & naive way, suppose I have 1 MB L2 cache set aside, should I create a window of 1 MB for my most frequently used data and set hitRatio as 1 or should I create a window of 2 MB and set hitRatio as 0.5? Why?

There is not enough information to give a specific answer to this question. The knowledge that you have decided to have 1MB L2 cache set aside is the first key piece of information, but the second key piece of information is how much data do you actually have to cache? In addition, expected access pattern matters. Lets cover several cases:

  1. 1MB cache set aside, 1MB window, hitRatio 1. This implies that you only have 1MB of data that you would like to cache with this mechanism. With a 1MB cache and a 1MB window, there isn't any reason to choose anything other than 1 for hitRatio. If you only have 1MB of data to cache, and you can afford to carve off 1MB of L2, this is a completely sensible choice. You're essentially guaranteeing that no other activity can "evict" this "protected" data, once it appears in the cache carve-out.

  2. 1MB cache set aside, 2MB window, hitRatio 0.5. This implies of course that you have at least 2MB of data that you would like to cache with this mechanism, so this is not directly comparable to case 1 above. The hitRatio of 0.5 can be thought of as a "guard" against thrashing. Let's consider several sub-cases:

    A. Suppose your 2MB of data is broken into 1MB regions A and B, and your code accesses all the data in region A (once), then all the data in region B (once), then all the data in region A (once), etc. If you chose a hit ratio of 1 with a 1MB cache carve out but a 2MB window, this setup would thrash. You'd fill the cache with region A, then evict and fill with region B, then evict and fill with region A, etc. Without a "hitRatio" mechanism/control, this kind of behavior for this kind of scenario would be unavoidable. So if you anticipate this kind of cyclic access patterns, a hit Ratio of 0.5 would be better (50% of your data access would benefit from L2 carve out, 50% would not) rather than having no benefit from the cache at all.

    B. Suppose your 2MB of data is accessed with high temporal locality. Your code access 1MB (say, region A) of the 2MB of data repeatedly, then accesses the other (region B) repeatedly. In this case, a carve-out might not be needed at all. If you wanted to use a carve-out, a hit ratio of 1 might make sense, because it means that this carve out is behaving more or less as ordinary cache, with the exception that the cacheable window is user-defined. In this example, your cache would fill up with your first 1MB of region A data, and then your code would benefit from cache as it reused that data. When your code switched modality and started to use the second 1MB of data (region B), this second 1MB would evict the first 1MB, and then as your code used the second 1MB repeatedly, it would derive again 100% benefit from the cache.

user_16959600
Related