Non-monotonic depth buffer; cyclical overlapping in OpenGL

user_5968729 0
Viewed 127

Is it possible to implement custom depth buffer values so that I can implement a non-transitive ordering? E.g.

Red > Green
Green > Blue
Blue > Red

Which I imagine would be implemented in the fragment shader or something, only writing the pixel if it's on top of the existing one as per the schema above. Note that since it's non-transitive it's impossible to assign a single numeric value to each of the colors and retain the ordering.

Put another way, I would like a custom z-buffer where each element is of a non-transitive type. So the z-buffer would be a large sheet full of Red, Green or Blue values. Assign a number to each and then compare as per the schema above to determine which one is on top. If all three colors are on the same fragment then I don't care which one is on top.

Image below illustrates what I mean (this is what I want it to be able to do):

enter image description here

(image from Wikipedia)

Is what I want here possible in OpenGL or am I going too much against the grain? If it is possible, how severe would the reduction in performance be?

depth-buffer glsl shader opengl
1 Answers

What you want to do is not only possible, it's relatively fast and simple. I've implemented something like this in my game in order to render the penrose triangle.

The trick is to use a stencil buffer instead of the depth buffer. There are actually several different ways you can use the stencil buffer to achieve a circular ordering effect, depending on your exact needs. I'll describe one possible algorithm here:

  1. Draw the red layer with no stencil test, while writing 1s to the stencil buffer.
  2. Draw the green layer with stencil test not equal to 1, while writing 2s to the stencil buffer. This will prevent it from being drawn where red has already been drawn, effectively putting green "behind" red.
  3. Draw the blue layer with stencil test not equal to 2. This will allow it to be drawn on top of red, but not green, effectively placing it between them.

The same technique generalizes to N layers like so:

- for each layer X going front-to-back, from `N - 1` down to `0`
    - draw layer X with stencil test `not equal to X + 1`, while writing `X` to the stencil buffer

Code for this might look something like:

glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
for (int X = N - 1; X >= 0; --X) {
    glStencilFunc(GL_NOTEQUAL, X + 1, 0xFF);
    // drawLayer(X);
}

This algorithm only guarantees that layer X will correctly interact with layers (X - 1) % N and (X + 1) % N. If you want more sophisticated rules, you will need a more sophisticated algorithm.

Note that with an 8-bit stencil buffer, you will be limited to 255 layers.

Also note that because this algorithm doesn't use the depth buffer, it works for 3D scenes as well - just enable the depth test like normal, and you're off to the races. That's how I'm able to render the penrose triangle example in 3D.

user_30647450
Related