How can Minecraft shader know about the objects when just pixels are the input

Viewed 180

I know that Minecraft Shaders are written in GLSL. Someone explained to me that it would get a set of pixel as an input and would return a bunch of new pixels. But how does the shader know about details like the positon of entitys, models, so on?

1 Answers

Describing a shader as something that "gets a set of pixels as an input and returns a bunch of new pixels" is a bit of an oversimplification. I don't know what is your knowledge of shaders, so I'll walk through several concepts relevant for your question.

The rendering process, called the rendering pipeline, is executed by the GPU and is split in several stages. For some of these stages you can implement a custom shader program. Thus each programmable rendering stage has its own shader whose inputs and outputs depend on the stage : https://www.khronos.org/opengl/wiki/Rendering_Pipeline_Overview

For Minecraft, you can find some information about the shader layout on the wiki : https://minecraft.fandom.com/wiki/Shaders

Here is also a page listing detailed resources about shader programming and Optifine's pipeline : https://wiki.shaderlabs.org/wiki/Getting_Started Be sure to check it as you can probably find most of the information you are looking for here.

I'll anyway try to give a brief summary about shaders and how they can "know" details about the scene (positions, entities...), more than just pixel colors.

Vertex and fragment shaders

As stated previously, there are several types of shader. The two "main" shaders that are executed during rendering are the vertex shader and the fragment shader (sometimes called the pixel shader). In Minecraft shaderpacks, they have respectively the extensions .vsh and .fsh.

The vertex shader is the first shader executed during the rendering, it takes vertices of the scene and attributes as inputs, and outputs new vertices. This is where you can implement transformations and displace vertices of a mesh for some effects.

The fragment shader is the last shader executed during rendering, and is the one generating the image to be put on the screen. It takes pixel coordinates as well as data calculated from the previous stages (vertex shader) and returns a color for each pixel.

Passing data from shader to shader

Attributes are special data that is passed to the GPU for each vertex and that you can use within a shader. You can for example have, for each vertex in the scene, an integer variable indicating to which object or entity this vertex belongs to, and use it in your vertex shader code. You can also generate new data that will then be passed from the vertex shader to the fragment shader input, these are special variables called varyings.

In GLSL, attributes and varyings are special variables you declare in your scripts. A vertex shader may contain :

// A variable that receives a value passed by the actual program
// on the CPU side.
attribute float someAttribute;

// A variable that you can fill with a value from this shader,
// and its content will be passed to the fragment shader.
varying vec3 someOutput;

Then in the fragment shader :

// Value of the variable with the same name from the vertex shader.
varying vec3 someOutput;

Each of the vertices have their own someAttribute and someOutput variable.

(You are maybe wondering how a value, assigned for each vertex, can then be passed to each pixels. I won't go into details, but what happens is that the value of someValue that the fragment shader receives for a pixel is an interpolation between the values of the someValue variable of the vertices of the triangle that has been projected on that pixel.)

Buffers

Another way a shader can have access to more information than just vertices and pixels (and that is probably more relevant for your question) is by using buffers. Often, rendering a scene to the screen is done in more than one pass. You may render shadows in one texture, then the geometry in another one, and finally combine these texture with another shader to create the final image to output to the screen.

In Minecraft Optifine shaderpacks for example, you can write a specific shaders to render an image in a buffer texture, where the color of each pixel represents some specific data you want to process independently. You can for example use a buffer where you render only the entity you are aiming at, coloring in white pixels that belong to the entity, and in black the other ones, and storing the result image in a texture. You can then access this mask texture from another shader to add some postprocess effects, for example highlighting the area of the entity you are aiming at. Accessing texture in a shader is simply done by using special functions such as texture() to get the pixel color at some coordinate in a texture which is referenced in some variable (sampler2D type).

As of how the shader choose what objects to render or not, this is decided on the CPU side. The program (here Minecraft) basically tells the GPU, via the OpenGL API, what 3D models to render, with which shader scripts, and if it must send the generated image directly to the screen or store it in a texture. The program can request the GPU to quickly switch between different preloaded rendering programs with different shaders and settings.

In Optifine, shader files for buffers correspond to the files starting with gdbuffers_. Taken directly from the doc (https://pastebin.com/aB5MJ7aN) :

These files are used to render terrain, entities, the sky, and almost everything else in the game. The specific name of the file tells you a bit more about what it's used to render. Skybasic runs first, and handles the main sky color. This is followed by skytextured, which handles the sun and moon. Up next comes terrain, which handles all opaque blocks.

On that same link, they give some example of how this can be used :

Create 2 buffers. one is a material buffer, the other is the translucent buffer. Make all transparent objects output their color to the translucent buffer, and a number representing their ID (passed in with varyings) to the material buffer. Composite [a type of shader defined by Optifine] can read the material buffer, and mix the translucent buffer with the opaque color buffer differently depending on the ID.

I hope this helps answering your question. I also recommend you to directly look into the source files of some shaderpacks to see how these are implemented.

Related