Image shadowing / rotating / flipping when applying Gaussian kernel in Fourier space

Question

DISCLAIMER UP FRONT: I know I can smooth/blur images using OpenCV, but my assignment is more under the hood so I can understand the process. I'm new to image processing and DFT's in general, so this is proving to be a challenge.

DESCRIPTION:

I'm doing some image processing using the FFTW library and OpenCV to read an image into an OpenCV::Mat object and grayscale color space. I convert the data to type double so I can create a double pointer to it which FFTW's function for my FFT requires. This is in a Mat object named imageDouble and the pointer to it is pImageDouble.

std::string filename = "";
Mat image = imread(filename, IMREAD_GRAYSCALE);

/***********************************************************************
*** creating Mat object of type *doube* and copying image contents to it
*** then creating a pointer to it.
***********************************************************************/
Mat imageDouble;
image.convertTo(imageDouble, CV_64FC1);
double* pImageDouble = imageDouble.ptr(0);

I also have a Gaussian kernel that I've read into a Mat object. I perform a circular shift where the center (and highest) value of the Gaussian kernel is shifted to the top left (the (0,0) position) of the kernel, and then I zero-pad the kernel to the size of the original image that I read in. The results (type double) are stored in a Mat object named paddedGKernel and the pointer is named pPaddedGKernel.

Mat paddedGKernel = padGKernelWithZeros(nRows, nCols, rolledGKernel);
double* pPaddedGKernel = paddedGKernel.ptr(0);

I initializefftw_complex objects to output the FFT results into, an fftw_plan, and then I allocate the memory for the fftw_complex objects and execute the plan. I use FFTW's fftw_plan_dft_r2c_2d() function to perform an FFT on both 2d objects (image and shifted/padded Gaussian kernel), then perform a pointwise multiplication in Fourier space to apply the Gaussian filter to the original image.

fftw_complex* out;               //for result of FFT for original image
fftw_complex* outGaussian;       //for result of FFT for Gaussian kernel
fftw_plan p;

/*************************************************
*** Allocating memory for the fftw_complex objects
*************************************************/
out = (fftw_complex*)fftw_malloc(sizeof(fftw_complex) * nRows * nCols);
outGaussian = (fftw_complex*)fftw_malloc(sizeof(fftw_complex) * nRows * nCols);


/****************************************
*** FFT on imageDouble, outputting to out
****************************************/
p = fftw_plan_dft_r2c_2d(imageDouble.rows, imageDouble.cols, pImageDouble, out, FFTW_ESTIMATE);
fftw_execute(p);


/**************************************************
*** FFT on paddedGKernel, outputting to outGuassian
**************************************************/
p = fftw_plan_dft_r2c_2d(paddedGKernel.rows, paddedGKernel.cols, pPaddedGKernel, outGaussian, FFTW_ESTIMATE);
fftw_execute(p);


/****************************************************
*** Pointwise multiplication to apply Gaussian kernel
****************************************************/
for (size_t i = 0; i < nCCols * nRows; i++)
{
    out[i][0] = out[i][0] * (1 / outGaussian[i][0]);
}

I then perform the inverse FFT fftw_plan_dft_c2r_2d() on the result of the pointwise multiplication and output that to an OpenCV::Mat object imageCloneDouble and convert the data back to uchar, placing the data in OpenCV::Mat imageBack.

/***********************************************************************
*** Making Mat object (type *double*) to put data into and pointer to it
***********************************************************************/
Mat imageCloneDouble = Mat::zeros(nRows, nCols, CV_64FC1);
double* pImageCloneDouble = imageCloneDouble.ptr(0);

/*****************************************************************************
*** Making and executing plan for inverse FFT, outputting to imageCloneDouble,
*** then normalizing since this function puts out unnormalized values.
*****************************************************************************/
fftw_plan pp = fftw_plan_dft_c2r_2d(imageCloneDouble.rows, imageCloneDouble.cols, out, pImageCloneDouble, FFTW_BACKWARD | FFTW_ESTIMATE);
fftw_execute(pp);
imageCloneDouble = imageCloneDouble / (nCols * nRows *2);

/***************************************************************************
*** Making Mat object (type *uchar*) and copying data inverse FFT data to it
***************************************************************************/
Mat imageBack;
imageCloneDouble.convertTo(imageBack, CV_8UC1);

When I show the transformed image imageBack I expect to get the original with the Gaussian kernel applied, but it looks like the original with potentially some modification overlaid with a rotated version of the original image that appears to have some modification. I can't understand where and why this flip/rotation is happening and why it is overlaid with what appears to be the original image, but I suspect it happens when I'm in Fourier space and performing a pointwise or element-wise multiplication. Either that or I'm omitting a process I need to perform on the result of the pointwise multiplication.

What do I need to do to this data in Fourier space to get back to the original image?

user_7328782 · Answer

This is your problem:

I perform a circular shift where the center (and highest) value of the Gaussian kernel is shifted to the top left (the (0,0) position) of the kernel, and then I zero-pad the kernel to the size of the original image that I read in.

You need to reverse these two operations: first pad to the right size, then apply the circular shift. You indeed need the center of the kernel to be at (0,0). But you need the kernel to be continuous in the circularly periodic world (i.e. replicating the image you should be able to see complete Gaussian kernels). When you pad after the circular shift, you separate the four quadrants of the kernel in this circularly periodic world.

A second problem is in this line:
out[i][0] = out[i][0] * (1 / outGaussian[i][0]);

First, why divide, not multiply? You are trying to apply a convolution, no?

Second, you are only multiplying the real component of the complex numbers, leaving the imaginary component unchanged. You need to do a full complex multiplication of two complex numbers to produce a new complex number. Cast the out pointer to std::complex*, then use the compiler's knowledge of complex arithmetic to do your bidding!

std::size_t N = nRows * nCols;
fftw_complex* out = fftw_alloc_complex(N);
fftw_complex* outGaussian = fftw_alloc_complex(N);

// ...

auto* out_p = reinterpret_cast*>(out);
auto* outGaussian_p = reinterpret_cast*>(outGaussian);
for (std::size_t i = 0; i < N; i++)
{
    out_p[i] *= outGaussian_p[i];
}

Note that std::complex and fftw_complex have the same memory layout, guaranteed by the C++ spec for std::complex, with the intention of being able to do this type of casting. No reinterpretation is actually done, only the C++ type system thinks so.

user_10755457 · Answer

The key to fixing this issue started with what Cris pointed out as the second problem. So the first thing to do was to implement the code provided in that part of the reply.

auto* out_p = reinterpret_cast*>(out);
auto* outGaussian_p = reinterpret_cast*>(outGaussian);
for (std::size_t i = 0; i < N; i++)
{
    out_p[i] *= outGaussian_p[i];
}

I had the Gaussian kernel normalized the usual way (dividing by the sum of all the values). After making the changes above from Cris, the image had the blur applied and the only issue was the range of pixel values had changed.

To fix that, I applied the formula for contrast stretching to match the original range and that solved the problem.

/* Finding the original image's minimum and maximum value */
double oMin, oMax;
minMaxLoc(image, &oMin, &oMax);

/* Finding min and max values for our 'imageBack' and rounding them. */
double min, max;
minMaxLoc(imageCloneDouble, &min, &max);
min = round(min);
max = round(max);

/* Code version of formula for contrast stretching */
if (oMin != min | oMax != max) {
    double numerator = oMax - oMin;
    double denominator = max - min;
    double scaledMin = -(min)*numerator;
    double scaledOMin = oMin * denominator;
    double innerTerm = scaledMin + scaledOMin;

    imageCloneDouble = ((numerator * imageCloneDouble) + innerTerm ) / denominator;
}

Image shadowing / rotating / flipping when applying Gaussian kernel in Fourier space

2 Answers