How to cluster image tiles of an array when considering the relative position of each tile?

I have a satellite image array (3D) which I tile into smaller arrays using Skimage to be later used for a segmentation task. My goal is to cluster the tiles into spatial groups based on their original relative position. I being by tiling the image:

tiles = skimage.util.view_as_windows(input_image, (256, 256, 3), step = 256)
tiled_image = tiles.reshape((-1,) + (256, 256, 3))

Before reshaping the tiles, I see the number of tiles produced in the X and Y direction. This lets me create an array which I fill with coordinates of the relative position of the tiles and then cluster to get a label:

from sklearn.cluster import KMeans
ycoord, xcoord = tiles.shape[0], tiles.shape[1]
print(ycoord, xcoord)

xrepeat = np.tile(np.arange(0,xcoord), ycoord)
yrepeat = np.tile(np.arange(0, ycoord), xcoord)

coords = pd.DataFrame(list(zip(xrepeat, yrepeat)),
               columns =['xcoord', 'ycoord'])
coords = coords.sort_values(by=['xcoord','ycoord'])
coords = coords.reset_index(drop=True)

kmeans = KMeans(init="random",n_clusters=5,n_init=10,max_iter=300,random_state=42)
kmeans.fit(coords)
labels = pd.DataFrame({'label': kmeans.labels_})
fold_labels = pd.concat([coords, labels], axis = 1)

My satellite image contains empty space which means that some tiles are also empty, therefore I need to remove the empty tiles. I'm doing this by determining if an elements in the image array have 0 values, or if there is no segmentation mask present:

def remove_empty(input_tiles):
    tile_indices = []
    empty = []
    for tile in range(input_tiles.shape[0]):
    #0:3 are RGB inputs, 3: are masks
        if np.any(np.sum(input_tiles[tile,:,:,0:3], axis=2) >= 1) and np.any(np.sum(input_tiles[tile,:,:,3:], axis=2) >= 1):
            tile_indices.append(tile)
            empty.append(1)
        else:
            empty.append(0)
            continue
    processed_tiles = input_tiles[tile_indices, :, :, :]
    return processed_tiles, tile_indices, empty

As a sense check I'm seeing where the tiles have been removed, using the indices from the XY coordinates array (clustered image), and it looks strange to me since the tiles removed don't look like they're following the distribution of empty pixel locations:

Any help looking over what I've done and what has gone wrong would be super helpful. So far all I can think of is that I've messed up making the array of tile coordinates (out of alignment so everything looks shifted?) or my criteria for empty tiles is not working as it should?

Thanks!