You may convert the image to HSV, apply threshold to find colored regions, and copy the result of cv2.threshold with cv2.THRESH_BINARY_INV only to the colored regions.
Main stages of the suggested solution:
- Convert from BGR to HSV color space, and get the saturation color channel.
all black and white are zero, and colored pixels are above zero.
- Apply threshold to the saturation channel.
- Find contours on the binarized saturation channel.
- Draw the contours as white (255 values) on black background to form a mask.
- Apply morphological closing for closing some black gaps.
- Get only the area inside the mask from
img_binary_inv (result of your code using cv2.THRESH_BINARY_INV).
- Copy masked
img_binary_inv to img_grey only in pixels that mask is white.
Complete code sample:
import numpy as np
import cv2
img_bgr = cv2.imread('img.jpg') # Read image as BGR
img_grey = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2GRAY) # Convert from BGR to grayscale.
thresh = 170
img_binary_inv = cv2.threshold(img_grey, thresh, 255, cv2.THRESH_BINARY_INV)[1] # Apply threshold and invert black/white
# Convert from BGR to HSV color space.
hsv = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2HSV)
# Get the saturation color channel - all black and white are zero, and colored pixels are above zero.
s = hsv[:, :, 1]
thresh = 100
s_binary = cv2.threshold(s, thresh, 255, cv2.THRESH_BINARY)[1] # Apply threshold to the saturation channel.
# Find contours on s_binary
cnts = cv2.findContours(s_binary, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)[-2] # Use index [-2] to be compatible to OpenCV 3 and 4
# Draw the contours as white (255 values) on black background.
mask = np.zeros_like(s_binary)
cv2.drawContours(mask, cnts, -1, 255, -1)
# Apply morphological closing for closing some black gaps.
mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, np.ones((3, 3)))
# Get only the area inside the mask from img_binary_inv
masked_binary_inv = cv2.bitwise_or(img_binary_inv, img_binary_inv, mask=mask)
# Copy masked_binary_inv to img_grey only in pixels that mask is white.
cv2.copyTo(masked_binary_inv, mask, img_grey)
cv2.imwrite('img_grey.png', img_grey) # Save result (as PNG and not JPEG for better quality).
# Show images
cv2.imshow('img_bgr', img_bgr)
cv2.imshow('s_binary', s_binary)
cv2.imshow('mask', mask)
cv2.imshow('masked_binary_inv', masked_binary_inv)
cv2.imshow('img_grey', img_grey)
cv2.waitKey()
cv2.destroyAllWindows()
Result (img_grey):

The result doesn't look so good, due to the relatively low quality of the input image.
Intermediate results:
s_binary:

mask:

masked_binary_inv:
