I have a picture were I want to change all white-ish pixels to grey, but only for a certain area of the image. Example picture, I just want to change the picture outside of the red rectangle, without changing the image within the red rectangle:

I already have the general code, which was part of someone elses Stackoverflow question, that changes the colour of every white pixel instead of only just the one outside of an area.
image = cv.imread("meme 2.jpg")
hsv = cv.cvtColor(image, cv.COLOR_BGR2HSV)
# Define lower and uppper limits of what we call "white-ish"
sensitivity = 19
lower_white = np.array([0, 0, 255 - sensitivity])
upper_white = np.array([255, sensitivity, 255])
# Mask image to only select white
mask = cv.inRange(hsv, lower_white, upper_white)
# Change image to grey where we found brown
image[mask > 0] = (170, 170, 170)
cv.imwrite(file, image)











