I am working on an isometric game engine and have already created an algorithm for pixel perfect click detection. Visit the project and notice that click detection is able to detect which edge of the tile was clicked. It is also checks the y-index to click the most upfront tile.

An Explanation of my current algorithm:

The isometric grid is made of tile images that are 100*65px. TileW=100, TileL=50, tileH=15

The map is represented by a three-dimensional array map[z][y][x].

Tile center points (x,y) are calculated like so:

//x, y, z are the position of the tile

if(y%2===0) { x-=-0.5; }    //To accommodate the offset found in even rows
this.centerX = (x*tileW) + (tileW/2);
this.centerY = (y*tileL) - y*((tileL)/2) + ((tileL)/2) + (tileH/2) - (z*tileH);

Prototype functions that determine if the mouse is within a given area on the tile:

Tile.prototype.allContainsMouse = function() {
    var dx = Math.abs(mouse.mapX-this.centerX),
        dy = Math.abs(mouse.mapY-this.centerY);

    if(dx>(tileW/2)) {return false;}    //Refer to image
    return (dx/(tileW*0.5) + (dy/(tileL*0.5)) < (1+tileHLRatio));
}

Tile.prototype.allContainsMouse() returns true if mouse is within green. Red area is cropped out by checking if dx > half the tile's width

Tile.prototype.topContainsMouse = function() {
    var topFaceCenterY = this.centerY - (tileH/2);
    var dx = Math.abs(mouse.mapX-this.centerX),
        dy = Math.abs(mouse.mapY-topFaceCenterY);

    return ((dx/(tileW*0.5) + dy/(tileL*0.5) <= 1));
};

Tile.prototype.leftContainsMouse = function() {
    var dx = mouse.mapX-this.centerX;
    if(dx<0) { return true; } else { return false; }
};

(If mouse is left of the center point)

Tile.prototype.rightContainsMouse = function() {
    var dx = mouse.mapX-this.centerX;
    if(dx>0) { return true; } else { return false; }
};

(If mouse is right of the center point)

Bringing all the methods together to work as one:

• Loop Through the entire 3d map[z][y][x] array
• if allContainsMouse() returns true, map[z][y][x] is the tile our mouse is on.
• Add this tile to the array tilesUnderneathMouse array.

• Loop through tilesUnderneathMouse array, and choose the tile with the highest y. It is the most upfront tile.

  if(allContainsMouse && !topContainsMouse)
  

• if(allContainsMouse && !topContainsMouse && leftContainsMouse)
  

(Similar concept applies for right)

Finally, my questions:

#1 How would you accomplish this, such that it is more efficient(not looping through all tiles)(pesudo code accepted)

#2 If you are unable to answer #1, what suggestions do you have to improve the efficiency of my click detection (chunk loading has already been considered)

What I've thought of:

I originally tried to solve this problem by not using tile center points, rather converting the mouse(x,y) position directly to the tile x,y. In my mind this is the hardest to code, yet most efficient solution. On a square grid it's very easy to convert an (x,y) position to a square on the grid. However in a staggered column grid, you deal with offsets. I tried to calculate offsets using the a function that takes an x or y value, and returns the resultant offset y, or x. The Zig-zag graph of arccos(cosx) solved that.

Checking if the mouse was within the tile, using this method was difficult and I couldn't figure it out. I was checking whether the mouse(x,y) was beneath a y=mx+b line that was dependent on the tileX, tileY approximation(a square grid approx).

If you got to here, Thanks!