I trained a CNN model using 3D kernels on an image dataset for the multilabel classification task. Training samples have no NaNs. Otherwise, exploding gradients will occur. Then I make predictions based on a new image by the pre-trained model; those missing values (empty area) in the image, as shown in the Fig below, are pixels that are not of interest:
So I don't need to do any data imputation on it. Ideally, pixels from that part should be predicted as a new label, e.g., '0' in my case since all classes that were trained were labelled from 1,2,3,4 etc.
before testing with 1D CNN, I made the data type of NaNs in the new image 'float', and then everything was fine in terms of those NaNs recognised as '0' by the model.
However, when testing with 3D CNN, the NaNs were predicted as different results depending on different scenarios (different number of inputs). For example:
I created a matrix equivalent of the input shape for the pre-trained 3D CNN, filled with NaNs:
patchesData = np.zeros((40, 21, 21, 26,3),dtype=np.float32)
patchesData[patchesData == 0] = 'nan'
Then fed into the pre-trained 3D CNN:
In: np.argmax(model.predict(patchesData), axis=1)
Out: array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int64)
This results in an array with 2 labels. When I reduced the number of inputs to 20, the predicted results come with:
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
dtype=int64)
I know I can easily mask out that missing part of the area that was wrongly predicted. I am just confused that why the model cannot handle NaNs consistently. What is the rationale behind this? Why 1D CNN has no problem with it?