Consider a full recangle (initially the whole picture) and take away one excluded box. You will get 2x2x2x2=16 possible rectangular subdivisions, for example this one.

  ┌────────────────────────┐
  │                        │
  │                        │
  ├───────┬───────┬────────┤
  │       │  exc  │        │
  │       │ lude  │        │
  │       ├───────┴────────┤
  │       │                │
  │       │                │
  └───────┴────────────────┘

For each box in the subdivision, take away the next excluded box. Do this N times, and take the biggest box of the final step.

Here's a potential solution to find the bounding box contour with the largest surface area. We have two requirements:

1. Largest bounding box is not intersecting with any other box
2. Largest bounding box is not inside another box

Essentially we can reword the two requirements to this:

1. Given C1 and C2, determine if C1 and C2 intersect
2. Given C1 and C2, check if there is a point from C1 in C2

To solve #1, we can create a contour_intersect function that uses a bitwise AND operation with np.logical_and() to detect intersection. The idea is to create two separate masks for each contour and then use the logical AND operation on them. Any points that have a positive value (1 or True) will be points of intersection. Essentially, if the entire array is False then there was no intersection between the contours. But if there is a single True, then the contours touched at some point and thus intersect.

For #2, we can create a function contour_inside and use cv2.pointPolygonTest() to determine if a point is inside, outside, or on the edge of a contour. The function returns +1, -1, or 0 to indicate if a point is inside, outside, or on the contour, respectively. We find the centroid of C1 and then check if that point is inside C2.

Here's an example to visualize the scenarios:

Input image with three contours. Nothing special here, the expected answer would be the contour with the largest area.

Answer:

Contour #0 is the largest

Next we add two additional contours. Contour #3 will represent the intersection scenario and contour #4 will represent the inside contour scenario.

Answer:

Contour #0 has failed test
Contour #1 has failed test
Contour #2 is the largest

To solve this problem, we find contours then sort using contour area from largest to smallest. Next, we compare this contour with all other contours and check the two cases. If either case fails, we dump the current contour and move onto the next largest contour. The first contour that passes both tests for all other contours is our largest bounding box contour. Normally, contour #0 would be our largest but it fails the intersection test. We then move onto contour #1 but this fails the inside test. Thus the last remaining contour that passes both tests is contour #2.

import cv2
import numpy as np

# Check if C1 and C2 intersect
def contour_intersect(original_image, contour1, contour2):
    # Two separate contours trying to check intersection on
    contours = [contour1, contour2]

    # Create image filled with zeros the same size of original image
    blank = np.zeros(original_image.shape[0:2])

    # Copy each contour into its own image and fill it with '1'
    image1 = cv2.drawContours(blank.copy(), contours, 0, 1)
    image2 = cv2.drawContours(blank.copy(), contours, 1, 1)

    # Use the logical AND operation on the two images
    # Since the two images had bitwise and applied to it,
    # there should be a '1' or 'True' where there was intersection
    # and a '0' or 'False' where it didnt intersect
    intersection = np.logical_and(image1, image2)

    # Check if there was a '1' in the intersection
    return intersection.any()

# Check if C1 is in C2
def contour_inside(contour1, contour2):
    # Find centroid of C1
    M = cv2.moments(contour1)
    cx = int(M['m10']/M['m00'])
    cy = int(M['m01']/M['m00'])

    inside = cv2.pointPolygonTest(contour2, (cx, cy), False)

    if inside == 0 or inside == -1:
        return False
    elif inside == 1:
        return True

# Load image, convert to grayscale, Otsu's threshold
image = cv2.imread('1.png')
original = image.copy()
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]

# Find contours, sort by contour area from largest to smallest
cnts = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
sorted_cnts = sorted(cnts, key=lambda x: cv2.contourArea(x), reverse=True)

# "Intersection" and "inside" contours
# Add both contours to test 
# --------------------------------
intersect_contour = np.array([[[230, 93]], [[230, 187]], [[326, 187]], [[326, 93]]])
sorted_cnts.append(intersect_contour)
cv2.drawContours(original, [intersect_contour], -1, (36,255,12), 3)

inside_contour = np.array([[[380, 32]], [[380, 229]], [[740, 229]], [[740, 32]]])
sorted_cnts.append(inside_contour)
cv2.drawContours(original, [inside_contour], -1, (36,255,12), 3)
# --------------------------------

# Find centroid for each contour and label contour number
for count, c in enumerate(sorted_cnts):
    M = cv2.moments(c)
    cx = int(M['m10']/M['m00'])
    cy = int(M['m01']/M['m00'])
    cv2.putText(original, str(count), (cx-5, cy+5), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (246,255,12), 3)

# Find largest bounding box contour
largest_contour_name = ""
largest_contour = ""
contours_length = len(sorted_cnts)
for i1 in range(contours_length):
    found = True
    for i2 in range(i1 + 1, contours_length):
        c1 = sorted_cnts[i1]
        c2 = sorted_cnts[i2]
        
        # Test intersection and "inside" contour
        if contour_intersect(original, c1, c2) or contour_inside(c1, c2):
            print('Contour #{} has failed test'.format(i1))
            found = False
            continue
    if found:
        largest_contour_name = i1
        largest_contour = sorted_cnts[i1]
        break
print('Contour #{} is the largest'.format(largest_contour_name))
print(largest_contour)

# Display
cv2.imshow('thresh', thresh)
cv2.imshow('image', image)
cv2.imshow('original', original)
cv2.waitKey()

Note: The assumption is that you have an array of contours from cv2.findContours() with the format like this example:

cnts = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
sorted_cnts = sorted(cnts, key=lambda x: cv2.contourArea(x), reverse=True)
for c in sorted_cnts:
    print(c)
    print(type(c))
    x,y,w,h = cv2.boundingRect(c)
    print((x,y,w,h))

Output

[[[230  93]]

 [[230 187]]

 [[326 187]]

 [[326  93]]]
<class 'numpy.ndarray'>
(230, 93, 97, 95)

Performance note: The intersection check function suffers on the performance side since it creates three copies of the input image to draw the contours and may be slower when it comes to execution time with a greater number of contours or a larger input image size. I'll leave this optimization step to you!

You can use the cv2.boundingRect() method to get the x, y, w, h of each bounding box, and with the x, y, w, h of each bounding box, you can use the condition x2 + w2 > x1 > x2 - w1 and y2 + h2 > y1 > y2 - h1 to check if any two bounding boxes intersect or are within each others:

import cv2
import numpy as np

def intersect(b1, b2):
    x1, y1, w1, h1 = b1
    x2, y2, w2, h2 = b2
    return x2 + w2 > x1 > x2 - w1 and y2 + h2 > y1 > y2 - h1

# Here I am generating a random array of 10 boxes in the format [[x,y], [x,y], [x,y], [x,y]]
np.random.seed(55)
boxes = np.random.randint(10, 150, (10, 4, 2)) + np.random.randint(0, 300, (10, 1, 2))

bounds = [cv2.boundingRect(box) for box in boxes]
valids = [b1 for b1 in bounds if not any(intersect(b1, b2) for b2 in bounds if b1 != b2)]
if valids:
    x, y, w, h = max(valids, key=lambda b: b[2] * b[3])
    print(f"x: {x} y: {y} w: {w} h: {h}")
else:
    print("All boxes intersect.")

Output:

x: 75 y: 251 w: 62 h: 115

For visualization:

import cv2
import numpy as np

def intersect(b1, b2):
    x1, y1, w1, h1 = b1
    x2, y2, w2, h2 = b2
    return x2 + w2 > x1 > x2 - w1 and y2 + h2 > y1 > y2 - h1

np.random.seed(55)
boxes = np.random.randint(10, 150, (10, 4, 2)) + np.random.randint(0, 300, (10, 1, 2))

bounds = [cv2.boundingRect(box) for box in boxes]
valids = [b1 for b1 in bounds if not any(intersect(b1, b2) for b2 in bounds if b1 != b2)]

img = np.zeros((500, 500), "uint8")
for x, y, w, h in bounds:
    cv2.rectangle(img, (x, y), (x + w, y + h), 255, 1)

if valids:
    x, y, w, h = max(valids, key=lambda b: b[2] * b[3])
    cv2.rectangle(img, (x, y), (x + w, y + h), 128, -1)

cv2.imshow("IMAGE", img)
cv2.waitKey(0)

Output:

Assumption: you want the largest box from your array that complies with your rules, and it is not the largest NEW bounding box that complies.

This is pseudo code, you still have to fill in blanks

int largestBoxIndex = -1;
int largestBoxArea = -1;

for (i=0; i<allBoxes[].length; i++)
{
    box CurrentBox =  allBoxes[i];
    bool isComply = false;
    for (j=0; j<allBoxes[].length; j++)
    {
        isComply = false;

        if(i==j) break;

        ComparedBox = allBoxes[j]

        if (isIntersected(CurrentBox, ComparedBox)) break;

        if (isInside(CurrentBox, ComparedBox)) break;
        
        isComply = true;    
    }

    if(isComply)
        if(Area(allBoxes[i]) > largestBoxArea)
        {
            largestBoxArea = Area(allBoxes[i]):
            largestBoxIndex =i;
        }

}

if(largestBoxIndex != -1)
    largestBoxIndex;//this is the largest box

A simple mathematical solution to the problem

Suppose you are given 5 rectangles as shown below:

rects = [[100, 100, 200, 200],
         [200, 200, 200, 200],
         [200, 500, 200, 200],
         [350,  50, 150, 200],
         [500, 400, 200, 300]]

Note that the format of these rectangles is: [x, y, width, height] Where, (x, y) is the coordinate of the top left corner of the rectangle, and width & height are the width and height of the rectangle respectively. You will have to covert your coordinates in this format first.

3 out of these 5 are intersecting.

Now what we will do is iterate over these rectangles one by one, and for each rectangle, find the intersection of this rectangle with the other rectangles one by one. If any rectangle is found to be intersecting with any of the other rectangles, then we'll set the flag value for the two rectangles as 0. If a rectangle is found not to be intersecting with any other rectangle, then its flag value will be set to 1. (Default flag value is -1). Finally, we'll find the rectangle of the greatest area among the rectangles with flag value 1.

Let's see the code for finding the intersection area of the two rectangles:

# Rect : [x, y, w, h]
def Intersection(Rect1, Rect2):
    x = max(Rect1[0], Rect2[0])
    y = max(Rect1[1], Rect2[1])
    w = min(Rect1[0] + Rect1[2], Rect2[0] + Rect2[2]) - x
    h = min(Rect1[1] + Rect1[3], Rect2[1] + Rect2[3]) - y
    
    if w < 0 or h < 0: 
        return None
    return [x, y, w, h]

This function will return None if there is no intersecting area between these rectangles or it will return the coordinates of the intersection rectangle(Ignore this value for the current problem. This might be helpful in other problems).

Now, let's have a look at the algorithm.

n = len(rects)

# -1 : Not determined
#  0 : Intersects with some
#  1 : No intersection
flag = [-1]*n

for i in range(n):
    if flag[i] == 0:
        continue

    isIntersecting = False
    for j in range(n):
        if i == j or flag[j] == 1:
            continue
        
        Int_Rect = Intersection(rects[i], rects[j])
        if Int_Rect is not None:
            isIntersecting = True
            flag[j] = 0
            flag[i] = 0
            break
    
    if isIntersecting == False:
        flag[i] = 1

# Finding the maximum area rectangle without any intersection.
maxRect = None
maxArea = -1

for i in range(n):
    if flag[i] == 1:
        if rects[i][2] * rects[i][3] > maxArea:
            maxRect = rects[i]
            maxArea = rects[i][2] * rects[i][3]

print(maxRect)

Note: Add the "excluded areas" rectangle coordinates to the rects list and assign their flag value as 0 to avoid them from getting selected as the maximum area rectangle.

This solution does not involve any images so it will be the fastest algorithm unless it is optimized.

Find the biggest square in numpy array

Maybe this would help? If you know the size of the whole area you can calculate the biggest box within numpy array. If you set all your given boxes to 1 and your whole area to 0 you need to find the largest area that is unique and not 1.

In other words:

1. rectangles that share points are excluded
2. of the remaining rectangles, take the largest

No need for contours, centroids, bounding boxes, masking or redrawing pixels!

As stated before, in the provided case, the rectangles coordinates contain duplicates. Here, we use a single class to store the outer limits of the rectangle. The Separating Axis theorem from this answer by @samgak is used in an intersects() method.

from __future__ import annotations # optional
from dataclasses import dataclass # optional ?

@dataclass
class Rectangle:
    left: int
    top: int
    right: int
    bottom: int
    def __repr__(self):
        """String representation of the rectangle's coordinates."""
        return f"⟔ {self.left},{self.top} ⟓ {self.right},{self.bottom}"
    def intersects(self, other: Rectangle):
        """Whether this Rectangle shares points with another Rectangle."""
        h = self.right < other.left or self.left > other.right
        v = self.bottom < other.top or self.top > other.bottom
        return not h or not v
    def size(self):
        """An indicator of the Rectangle's size, equal to half the perimeter."""
        return self.right - self.left + self.bottom - self.top

main = Rectangle(100, 100, 325, 325)

others = {
    0: Rectangle(100, 100, 400, 400),
    1: Rectangle(200, 200, 300, 300),
    2: Rectangle(200, 300, 300, 500),
    3: Rectangle(300, 300, 500, 500),
    4: Rectangle(500, 500, 600, 600),
    5: Rectangle(350, 350, 600, 600),
}

for i, r in others.items():
    print(i, main.intersects(r), r.size())

Simply put, h is True if the other rectangle is completely to the left or to the right; v is True if it's at the top or the bottom. The intersects() method returns True if the rectangles share points (even so much as a corner).

Output:

0 True 600
1 True 200
2 True 300
3 True 400
4 False 500
5 False 200

It is then trivial to find the largest:

valid = {r.size():i for i, r in others.items() if not main.intersects(r)}
print('Largest:', valid[max(valid)], 'with size', max(valid))

Output:

Largest: 4 with size 500

This answer assumes left < right and top < bottom for all rectangles.

The following function turns the provided rectangle coordinates to the kind used by the Rectangle class above. This assumes that the order is [[l, t], [r, t], [r, b], [l, b]] (a path).

def trim(coordinates):
    """Remove redundant coordinates in a path describing a rectangle."""
    return coordinates[0][0], coordinates[1][1], coordinates[2][0], coordinates[3][1]

Finally, we want to do this for all rectangles, not just a "main" one. We can simply have each rectangle be the main one in turns. Use itertools.combinations() on an iterable such as a list:

itertools.combinations(rectangles, 2)

This will ensure that we don't compare two rectangles more than one time.

Here's a O(n^2) solution. find_maxbox takes array of rectangles and convert them into Box objects and then compare each pair of boxes to eliminate invalid rectangles. This solution assumes that the rectangles' sides are parallel to X-Y axes.

class Box():
    def __init__(self, coordinates):
        self.coordinates = tuple(sorted(coordinates))
        self.original = coordinates
        self.height = abs(self.coordinates[0][1] - self.coordinates[3][1])
        self.width = abs(self.coordinates[0][0] - self.coordinates[3][0])
        self.excluded = False

    def __eq__(self, b2):
        return self.coordinates == b2.coordinates

    def get_area(self):
        return self.height * self.width

    def bounding_box(self, b2):
        maxX, maxY = map(max, zip(*self.coordinates, *b2.coordinates))
        minX, minY = map(min, zip(*self.coordinates, *b2.coordinates))
        return Box([(minX, minY), (maxX, minY), (minX, maxY), (maxX, maxY)])

    def intersects(self, b2):
        box = self.bounding_box(b2)
        if box.height < self.height + b2.height and box.width < self.width + b2.width:
            return True
        else: return False

    def encloses(self, b2):
        return self == self.bounding_box(b2)

    def exclude(self):
        self.excluded = True

    def is_excluded(self):
        return self.excluded

    def __str__(self):
        return str(self.original)

    def __repr__(self):
        return str(self.original)

# Pass array of rectangles as argument.
def find_maxbox(boxes):
    boxes = sorted(map(Box, boxes), key=Box.get_area, reverse=True)
    _boxes = []
    _boxes.append((boxes[0], boxes[0]))
    for b1 in boxes[1:]:
        b2, bb2 = _boxes[-1]
        bbox = b1.bounding_box(bb2)
        if not b1.intersects(bb2):
            _boxes.append((b1, bbox))
            continue
        for (b2, bb2) in reversed(_boxes):
            if not b1.intersects(bb2):
                break
            if b1.intersects(b2):
                if b2.encloses(b1):
                    b1.exclude()
                    break
                b1.exclude()
                b2.exclude()
        _boxes.append((b1, bbox))

    for box in boxes:
        if box.is_excluded():
            continue
        else: return box.original

    return None