I find the visibility algorithm in one great website
Since it is quiet long ago article, I am not sure that the author will give the answer.
As you can see, the interactive playground(try it yourself) does not work well in right upper side, especially overlapping edges.

// Pseudocode of main algorithm
var endpoints; # list of endpoints, sorted by angle
var open = []; # list of walls the sweep line intersects
loop over endpoints:
remember which wall is nearest
add any walls that BEGIN at this endpoint to 'walls'
remove any walls that END at this endpoint from 'walls'
figure out which wall is now nearest
if the nearest wall changed:
fill the current triangle and begin a new one
Here is also part of working JavaScript code
// Main algorithm with working Javascript code
export const calculateVisibility = (origin, endpoints) => {
let openSegments = [];
let output = [];
let beginAngle = 0;
endpoints.sort(endpointCompare);
for(let pass = 0; pass < 2; pass += 1) {
for (let i = 0; i < endpoints.length; i += 1) {
let endpoint = endpoints[i];
let openSegment = openSegments[0];
if (endpoint.beginsSegment) {
let index = 0
let segment = openSegments[index];
while (segment && segmentInFrontOf(endpoint.segment, segment, origin)) {
index += 1;
segment = openSegments[index]
}
if (!segment) {
openSegments.push(endpoint.segment);
} else {
openSegments.splice(index, 0, endpoint.segment);
}
} else {
let index = openSegments.indexOf(endpoint.segment)
if (index > -1) openSegments.splice(index, 1);
}
if (openSegment !== openSegments[0]) {
if (pass === 1) {
let trianglePoints = getTrianglePoints(origin, beginAngle, endpoint.angle, openSegment);
output.push(trianglePoints);
}
beginAngle = endpoint.angle;
}
}
}
return output;
};
In my opinion, it happens since the algorithm thinks right edge is shown whole part when it is the closest edge although it can be hidden from whole edges.
I think I can modify the algorithm to use the current closest edge and newer current closest edge but not sure...
How should I fix it?