Trying to synchronize increment of a counter SSBO in compute shader

Viewed 168

I am pretty new to Vulkan and am working off of nvpro samples vk mini path tracer and am sending a counter variable initialized to 0 to my compute shader as a SSBO. I am able to increment the value in the shader and reinterpret the value back on the CPU. However, the value is lower than I expected and returns a different number (around 30-60) each time. I am having trouble figuring out how to synchronize the buffer value and I think it may have to deal with the parallelization of the work groups in the vkCmdDispatch function call.

Is there a way to increment this counter and have that be reflected in all other shader invocations? Where do I have to setup this synchronization? Shader code or CPU side?

I’ve tried looking into GLSL’s memoryBarrierBuffer() as well as other memory barrier concepts but I can’t tell if those apply to the single compute shader with workgroup dispatches.

render.cpp:

const uint64_t rayCount = 0;
nvvk::Buffer genRayCount;
 
...

VkCommandBuffer uploadCmdBuffer = AllocateAndBeginOneTimeCommandBuffer(context, cmdPool);
       // We get these buffers' device addresses, and use them as storage buffers and build inputs.
       const VkBufferUsageFlags usage = VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT;
       const VkMemoryPropertyFlags memUsage = VK_MEMORY_PROPERTY_HOST_CACHED_BIT 
| VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT 
| VK_MEMORY_PROPERTY_HOST_COHERENT_BIT;
      
       genRayCount = allocator.createBuffer(uploadCmdBuffer, sizeof(uint64_t), &rayCount, usage, memUsage);
 
... // other bindings
 
descriptorSetContainer.addBinding(3, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1, VK_SHADER_STAGE_COMPUTE_BIT);
 
// descriptor sets
std::array<VkWriteDescriptorSet, 4> writeDescriptorSets;
...
// binding 3 for ray gen counter
   VkDescriptorBufferInfo genRayCountDescriptorBufferInfo{};
   genRayCountDescriptorBufferInfo.buffer   = genRayCount.buffer;
   genRayCountDescriptorBufferInfo.range    = genRayCountSizeBytes;
   writeDescriptorSets[3] = descriptorSetContainer.makeWrite(0, 2, &genRayCountDescriptorBufferInfo);
 
...
vkCmdDispatch(cmdBuffer, (uint32_t(cam.width) + workgroup_width - 1) / workgroup_width,
                   (uint32_t(cam.height) + workgroup_height - 1) / workgroup_height, 1);
...

raytrace.comp.glsl:

...
layout(binding = 3, set = 0) buffer generatedRayCount
{
 uint genRayCount;
};
...
void main()
{
 // The resolution of the buffer, which in this case is a hardcoded vector
 // of 2 unsigned integers:
 const uvec2 resolution = uvec2(cam.render_width, cam.render_height);
 // Get the coordinates of the pixel for this invocation:
 //
 // .-------.-> x
 // |       |
 // |       |
 // '-------'
 // v
 // y
 uvec2 pixel = gl_GlobalInvocationID.xy;
 
 // If the pixel is outside of the image, don't do anything:
 if((pixel.x >= resolution.x) || (pixel.y >= resolution.y))
 {
   return;
 }
  // Get the index of this invocation in the buffer:
 uint linearIndex = resolution.x * pixel.y + pixel.x;
 
  uint frame_size = resolution.x * resolution.y;
 
 uint count = genRayCount;

 if(rayBuffer[linearIndex].isIntersected) {
   //... generate a 2nd ray
   // increment ray counter
  
   genRayCount += 1;
  
 }
 memoryBarrierBuffer();
  const vec3 pixelColor = (rayBuffer[linearIndex].isIntersected) ? vec3(0.9) :
                           vec3(float(pixel.x) / resolution.x, 
                           float(count) / resolution.x,                            float(pixel.y) / resolution.y);
  // Write the color to the buffer.
 imageData[linearIndex] += pixelColor;
}

1 Answers

You likely want to use atomic add here.

that += operation is causing undefined behaviour. Imagine you have 500 threads labeled 0-499. Assume the current value of your counter is 10. Thread 32 reads 10, so does thread 411. They both try to write 11 to your variable, causing an under count. You alos likely have cases of threads overwriting higher count values with lowers and all sorts of multithreading data corruption.

You need to sync your threads for this to work.

Related