Object detection- How to re-apply manipulated pixels within co-ordinates to the original image to highlight RoI

Viewed 256

I have a set of coordinates for an RGB image (1920 x 1080) stored as a list of list which is RoI (region of interests) as follows

[[1538, 234, 1626, 313],
[421, 231, 472, 288],
[918, 199, 988, 270],
[802, 0, 963, 114],
[1732, 0, 1909, 108]]

I then apply the following conditions to change specific pixel values within the co-ordinates in the list of the list above only to highlight these RoIs by changing the pixels into white and black.

for coords in list_of_coords:

            roi = orig[coords[1]:coords[3], coords[0]:coords[2]]

            roi [(roi > 200)] = 0
            roi [(roi < 200)] = 255

How can I now take these new modified pixel values and apply them on top of the original image with a degree of transparency? Ideally, the rest of the image will be the same whilst the values within these coordinates will have a transparent mask on top of them.

Here is an example of how the desired output should look like. I have taken this from Mask RCNN, where within the rectangle coordinates, a slight shade of blue is added to highlight the RoI. The rectangle boxes are represented with the list of list above, and the shade of blue is the manipulation from the conditions mentioned above.

enter image description here

2 Answers

I think the answer you are looking for is alpha blending. If you have the two images (of the same size) that you gave in description (let's call them img and annotation), you can do :

alpha = 0.5 
# How much you want img to be opaque, alpha = 1 leads to your result being img,
# alpha = 0 to result==annotation
# with alpha = 0.5, you will get a blending with each image have the same 
# importance
result = cv2.addWeighted(img, alpha, annotation, 1-alpha, 0)

You can find some explanation about of the function here.

Now, if your images are not of the same size, you need to resize them before blending them (I would resize the smaller one to get the same size as the bigger one), so it should look something like this :

width = img.shape[1]
height = img.shape[0]
dim = (width, height)
annotation = cv2.resize(annotation, dim) 

I'm still not sure, if I fully understood, what you want to achieve, but here's my attempt to answer your question:

Basically, for each set of coordinates, you'd need to do the following:

  1. Get a ROI cutout of your original image. (You already have that.)
  2. Generate some binary mask (with respect to some metric or whatever information you have) to indicate which part inside the ROI you want to manipulate, e.g. the car shown in your image. In my code below, I chose to mask any pixels with some high blue channel value.
  3. Generate a blend image with the same size of the ROI and fill all pixels with a solid color with respect to the generated mask. In my code below, I chose to set up some solid green.
  4. Use OpenCV's addWeighted method for the alpha blending.
  5. Copy back the updated ROI cutout to your original image.

Here's the final output for my example image and coordinates:

Final output

That's my code:

import cv2
from matplotlib import pyplot as plt
import numpy as np

# Load image
orig = cv2.imread('path/to/your/image.png')
orig_back = orig.copy()

# Set up list of list of coordinates (i.e. rectangular ROIs)
list_of_coords = [[90, 0, 260, 30],
                  [60, 180, 100, 250],
                  [300, 300, 380, 370]]

# Set up alpha for alpha blending
alpha = 0.5

for coords in list_of_coords:

    # Get ROI cutout of image
    roi = orig_back[coords[1]:coords[3], coords[0]:coords[2]]
    roi_back = roi.copy()

    # Generate some mask; examplary here: Mask anything with some high
    # blue channel value
    mask = roi[:, :, 0] > 172

    # Generate blend image; exemplary here: Blend with solid green
    blend = np.zeros_like(roi)
    blend[mask, 1] = 255

    # Alpha blending original image with blend image
    roi[mask, :] = cv2.addWeighted(roi[mask, :], alpha,
                                   blend[mask, :], 1 - alpha,
                                   0)

    # Copy updated ROI cutout back to image
    orig[coords[1]:coords[3], coords[0]:coords[2]] = roi

    # Add rectangle to image for location
    orig = cv2.rectangle(orig,
                         (coords[0], coords[1]),
                         (coords[2], coords[3]),
                         (0, 255, 0),
                         2)

    # Some visualization output for better understanding
    plt.figure(0, figsize=(16, 8))
    plt.subplot(221)
    plt.imshow(cv2.cvtColor(roi_back, cv2.COLOR_BGR2RGB))
    plt.title('ROI cutout of image')
    plt.subplot(222)
    plt.imshow(mask, cmap='gray')
    plt.title('Masked portion of ROI with high blue channel value')
    plt.subplot(223)
    plt.imshow(cv2.cvtColor(blend, cv2.COLOR_BGR2RGB))
    plt.title('Blend image, solid green')
    plt.subplot(224)
    plt.imshow(cv2.cvtColor(roi, cv2.COLOR_BGR2RGB))
    plt.title('Updated ROI cutout of image')
    plt.tight_layout()
    plt.show()


# Final output
cv2.imshow('Original image with updates and rectangles', orig)
cv2.waitKey(0)
cv2.destroyAllWindows()

And, here's the additional visualization for the first ROI:

Additional visualization

Hopefully, that's what you're looking for!

----------------------------------------
System information
----------------------------------------
Platform:    Windows-10-10.0.16299-SP0
Python:      3.8.5
Matplotlib:  3.3.1
NumPy:       1.19.1
OpenCV:      4.4.0
----------------------------------------
Related