moviepy black border around png when compositing into an MP4

user_1552942 0
Viewed 698

compositing a png into an MP4 video creates a black border around the edge.

This is using moviepy 1.0.0

Code below reproduces the MP4 with the attached red text png.

enter image description here

import numpy as np
import moviepy.editor as mped
def composite_txtpng_on_colour():
    bg_color = mped.ColorClip(size=[400, 300], color=np.array([0, 255, 0]).astype(np.uint8),
                          duration=2).set_position((0, 0))
    text_png_postition = [5, 5]
    text_png = mped.ImageClip("./txtpng.png", duration=3).set_position((text_png_postition))

    canvas_size = bg_color.size
    stacked_clips = mped.CompositeVideoClip([bg_color, text_png], size=canvas_size).set_duration(2)
    stacked_clips.write_videofile('text_with_black_border_video.mp4', fps=24)

composite_txtpng_on_colour()

The result is an MP4 that can be played in VLC player. A screenshot of the black edge can be seen below:-

enter image description here

Any suggestions to remove the black borders would be much appreciated.

Update: It looks like moviepy does a blit instead of alpha compositing.

def blit(im1, im2, pos=None, mask=None, ismask=False):
    """ Blit an image over another.  Blits ``im1`` on ``im2`` as position ``pos=(x,y)``, using the
    ``mask`` if provided. If ``im1`` and ``im2`` are mask pictures
    (2D float arrays) then ``ismask`` must be ``True``.
    """
    if pos is None:
        pos = [0, 0]

    # xp1,yp1,xp2,yp2 = blit area on im2
    # x1,y1,x2,y2 = area of im1 to blit on im2
    xp, yp = pos
    x1 = max(0, -xp)
    y1 = max(0, -yp)
    h1, w1 = im1.shape[:2]
    h2, w2 = im2.shape[:2]
    xp2 = min(w2, xp + w1)
    yp2 = min(h2, yp + h1)
    x2 = min(w1, w2 - xp)
    y2 = min(h1, h2 - yp)
    xp1 = max(0, xp)
    yp1 = max(0, yp)

    if (xp1 >= xp2) or (yp1 >= yp2):
        return im2

    blitted = im1[y1:y2, x1:x2]

    new_im2 = +im2

    if mask is None:
        new_im2[yp1:yp2, xp1:xp2] = blitted
    else:
        mask = mask[y1:y2, x1:x2]
        if len(im1.shape) == 3:
            mask = np.dstack(3 * [mask])
        blit_region = new_im2[yp1:yp2, xp1:xp2]
        new_im2[yp1:yp2, xp1:xp2] = (1.0 * mask * blitted + (1.0 - mask) * blit_region)
    
    return new_im2.astype('uint8') if (not ismask) else new_im2

and so, Rotem is right.

new_im2[yp1:yp2, xp1:xp2] = (1.0 * mask * blitted + (1.0 - mask) * blit_region)

is

(alpha * img_rgb + (1.0 - alpha) * bg)

and this is how moviepy composites. And this is why we see black at the edges.

moviepy ffmpeg python-3.x
2 Answers

The main issue is the YUV420 color sub-sumpling, but it's also a result of compression artifacts, and imperfect "Text red" image.

The image imperfection is just in the alpha channel.
There are alpha (transparency) values that are not 255 and not 0 (semi-transparnt) pixels around the text.

The following code sample corrects it, and show the difference (using OpenCV):

orig_img = cv2.imread('txtpng.png', cv2.IMREAD_UNCHANGED)
img = orig_img.copy()
img[(img != 255) & (img != 0)] = 255  # Keep only two values: 0 and 255
cv2.imwrite('txtpng2.png', img)  # Write img to txtpng2.png
cv2.imshow('alpha diff', cv2.absdiff(orig_img[:,:,3], img[:,:,3])*100) # Show the difference in alpha channels
cv2.waitKey()
cv2.destroyAllWindows()

The above code keeps only two values: 0 and 255 in img.

Result ('alpha diff'):
enter image description here
As you can see there are differences.

Setting codec parameters:

For reference I created an uncompressed AVI video file:

# Save uncompressed AVI as reference
stacked_clips.write_videofile('text_with_black_border_video.avi', fps=24, codec='rawvideo', ffmpeg_params=['-pix_fmt', 'bgr24'])

I also tried to select H.264 codec with yuv444p pixel format, but for some reason it's not working.
I have selected H.265 codec instead.
Using ffmpeg_params, I also set '-crf', '10' for almost lossless video compression:

stacked_clips.write_videofile('text_with_black_border_video.mp4', fps=24, codec='libx265', ffmpeg_params=['-pix_fmt', 'yuv444p', '-crf', '10'])

Here is the complete code sample:

import numpy as np
import moviepy.editor as mped
import cv2

orig_img = cv2.imread('txtpng.png', cv2.IMREAD_UNCHANGED)
img = orig_img.copy()
img[(img != 255) & (img != 0)] = 255  # Keep only two values: 0 and 255
cv2.imwrite('txtpng2.png', img)

cv2.imshow('img', img)
cv2.imshow('alpha diff', cv2.absdiff(orig_img[:,:,3], img[:,:,3])*100) # Show the difference in alpha channels
cv2.waitKey()
cv2.destroyAllWindows()

def composite_txtpng_on_colour():
    bg_color = mped.ColorClip(size=[400, 300], color=np.array([0, 255, 0]).astype(np.uint8),
                          duration=2).set_position((0, 0))
    text_png_postition = [5, 5]
    text_png = mped.ImageClip('txtpng2.png', duration=3).set_position((text_png_postition))

    canvas_size = bg_color.size
    stacked_clips = mped.CompositeVideoClip([bg_color, text_png], size=canvas_size).set_duration(2)
    stacked_clips.write_videofile('text_with_black_border_video.mp4', fps=24, codec='libx265', ffmpeg_params=['-pix_fmt', 'yuv444p', '-crf', '10'])

    # Save uncompressed AVI as reference
    stacked_clips.write_videofile('text_with_black_border_video.avi', fps=24, codec='rawvideo', ffmpeg_params=['-pix_fmt', 'bgr24'])

composite_txtpng_on_colour()

Result (text_with_black_border_video.mp4):
enter image description here

Reference (uncompressed text_with_black_border_video.avi):
enter image description here

Magnified part:
enter image description here

Note:

  • I am using moviepy version 1.03

I figured out why H.264 codec is not working with pixel format yuv444p.

We can add '-report' to the list of ffmpeg_params:

stacked_clips.write_videofile('text_with_black_border_video.mp4', fps=24, codec='libx264', ffmpeg_params=['-pix_fmt', 'yuv444p', '-crf', '10', '-report'])

The logged report begins with:

... \\lib\\site-packages\\imageio_ffmpeg\\binaries\\ffmpeg-win64-v4.2.2.exe" -y -loglevel error -f rawvideo -vcodec rawvideo -s 400x300 -pix_fmt rgb24 -r 24.00 -an -i - -vcodec libx264 -preset medium -pix_fmt yuv444p -crf 10 -report -pix_fmt yuv420p text_with_black_border_video.mp4

The log shows that moviepy added -pix_fmt yuv420p argument.
pix_fmt argument is added twice: -pix_fmt yuv444p -pix_fmt yuv420p.
The yuv420p argument "wins".

Update:

Way to keep the edges slightly blurred:

The edges color is not back, but dark green.
The edges color is a result of Alpha compositing in RGB color space.
I suppose the compositing is performed using the simple formula:

dst_img = alpha*img_rgb + (1-alpha)*bg
Where alpha = img[:,:,0:3]/255.

When applying the above formula we are getting the following image:

enter image description here
Text edge color is dark green.

Suggested solution:
Apply alpha compositing in LAB color space.

Advantage:

Unlike the RGB and CMYK color models, CIELAB is designed to approximate human vision.

Linear operations in LAB color space are precepted as (approximate) linear by human vision.

Here is a code sample for alpha compositing in LAB color space:

img = cv2.imread('txtpng.png', cv2.IMREAD_UNCHANGED)
img2 = np.zeros((300, 400, 4), np.uint8)
img2[(300-img.shape[0])//2:(300+img.shape[0])//2, (400-img.shape[1])//2:(400+img.shape[1])//2, :] = img
alpha = img2[:, :, 3].astype(np.float64)/255  # Convert alpha to range [0, 1]
alpha = np.dstack((alpha, alpha, alpha))  # Duplicate alpha to 3 channels
img2 = img2[:, :, 0:3]  # Only BGR without alpha

bg = np.full_like(img2, (0, 255, 0)) # Green background

bg = cv2.cvtColor(bg, cv2.COLOR_BGR2LAB)
img2 = cv2.cvtColor(img2, cv2.COLOR_BGR2LAB)
composed_img = (img2.astype(np.float64)*alpha + bg.astype(np.float64)*(1-alpha)).astype(np.uint8)
composed_img = cv2.cvtColor(composed_img, cv2.COLOR_LAB2BGR)

cv2.imwrite('composed_img.png', composed_img) # Store the result

Result:
enter image description here
Text edge color looks better.

Note:
I couldn't find any papers about alpha compositing in LAB color space (but I didn't look so hard).

user_49267570

Don't use on_color to make a background color for your CompositeVideoClip, or else, edges will be black. Use this helper function to generate a color clip instead:

def get_color_video_clip(color, size, duration):
    image = Image.new("RGB", size, color)
    image_array = numpy.array(image)

    return mpy.ImageClip(image_array, duration=duration)

Also, as a sanity check, play the video back in different media players. On Mac, IINA made edges gray for me, while QuickTime Player had no such issues.

user_31516750
Related