I know the title isn't very clear but its kinda a complicated idea and I don't have the slightest clue how else to explain it. If you can think of a better title please, please, please , tell me and ill edit the title.
My problem is this: One of my close coder pals challenged me to re-create this Image to Image translation example. There are a few catches though.
- I cant use any pre-trained neural networks
- I have to make it run realtime (use webcam)
So far I have made it up to having the face swapped but i need to make it go back on to the non webcam image but there is an issue. to do that I have to re-build and warp the whole image around that image including filling the background behind the face that isn't included in the the original source image. I have tried to use impainting but on some occasions it takes part of the hair and neck and merges it into the background just creating a mess of skin and hair colors. I have also tried expanding my mask on the cv2 impainting function but that results in a big background color square that also looks terrible. I assume the best solution would be to segment off the biggest area around the head with some sort of segmenting algorithm, then clone part of that area to preserve the background texture instead of creating a new bad texture and placing that cloned area inside the mask. This all has to be done in realtime.
In the end I need to copy the person in img1, then remove them (part im stuck on). then take the facial landmarks from img2 and map those landmarks onto img1's clone's face, then add the re-mapped img1 back onto the img1 background. I know i cant communicate these ideas as clearly as someone more qualified as i am in 8th grade so if any clarification is required please ask.