OpenGL Terrain System, small height difference between GPU and CPU

A quick summary:

I've a simple Quad tree based terrain rendering system that builds terrain patches which then sample a heightmap in the vertex shader to determine the height of each vertex.

The exact same calculation is done on the CPU for object placement and co.

Super straightforward, but now after adding some systems to procedurally place objects I've discovered that they seem to be misplaced by just a small amount. To debug this I render a few crosses as single models over the terrain. The crosses (red, green, blue lines) represent the height read from the CPU. While the terrain mesh uses a shader to translate the vertices.

(I've also added a simple odd/even gap over each height value to rule out a simple offset issue. So those ugly cliffs are expected, the submerged crosses are the issue)

I'm explicitly using GL_NEAREST to be able to display the "raw" height value:

As you can see the crosses are sometimes submerged under the terrain instead of representing its exact height.

The heightmap is just a simple array of floats on the CPU and on the GPU.

How the data is stored

A simple vector<float> which is uploaded into a GL_RGB32F GL_FLOAT buffer. The floats are not normalized and my terrain usually contains values between -100 and 500.

How is the data accessed in the shader

I've tried a few things to rule out errors, the inital:

vec2 terrain_heightmap_uv(vec2 position, Heightmap heightmap)
{
    return (position + heightmap.world_offset) / heightmap.size;
}

float terrain_read_height(vec2 position, Heightmap heightmap)
{
    return textureLod(heightmap.heightmap, terrain_heightmap_uv(position, heightmap), 0).r;
}

Basics of the vertex shader (the full shader code is very long, so I've extracted the part that actually reads the height):

void main()
{
    vec4 world_position = a_model * vec4(a_position, 1.0);
    vec4 final_position = world_position;

    // snap vertex to grid
    final_position.x = floor(world_position.x / a_quad_grid) * a_quad_grid;
    final_position.z = floor(world_position.z / a_quad_grid) * a_quad_grid;

    final_position.y = terrain_read_height(final_position.xz, heightmap);

    gl_Position = projection * view * final_position;
}

To ensure the slightly different way the position is determined I tested it using hardcoded values that are identical to how C++ reads the height:

return texelFetch(heightmap.heightmap, ivec2((position / 8) + vec2(1024, 1024)), 0).r;

Which gives the exact same result...

How is the data accessed in the application

In C++ the height is read like this:

inline float get_local_height_safe(uint32_t x, uint32_t y)
{
    // this macro simply clips x and y to the heightmap bounds
    // it does not interfer with the result
    BB_TERRAIN_HEIGHTMAP_BOUND_XY_TO_SAFE;

    uint32_t i = (y * _size1d) + x;

    return buffer->data[i];
}

inline float get_height_raw(glm::vec2 position)
{
    position = position + world_offset;

    uint32_t x = static_cast<int>(position.x);
    uint32_t y = static_cast<int>(position.y);

    return get_local_height_safe(x, y);
}

float BB::Terrain::get_height(const glm::vec3 position)
{
    return heightmap->get_height_raw({position.x / heightmap_unit_scale, position.z / heightmap_unit_scale});
}

What have I tried:

• Comparing the Buffers
  I've dumped the first few hundred values from the vector. And compared it with the floating point buffer uploaded to the GPU using Nvidia Nsight, they are equal, rounding/precision errors there.
• Sampling method
  I've tried texture, textureLod and texelFetch to rule out some issue there, they all give me the same result.
• Rounding
  The super strange thing, when I round all the height values. They are perfectly aligned which just screams floating point precision issues.
• Position snapping
  I've tried rounding, flooring and ceiling the position, to ensure the position always maps to the same texel. I also tried adding an epsilon offset to rule out a positional precision error (probably stupid because the terrain is stable...)
• Heightmap sizes
  I've tried various heightmaps, also of different sizes.
• Heightmap patterns
  I've created a heightmap containing a pattern to ensure the position is not just offsetet.