Autoencoder to convert 2d to 3d layers ambiguity

Question

i got around some references and research papers and taking idea from one of them i thought to go ahead and implement the same the image reference-

So, here we are inputing a 2d input and the model outputs a 3d model of the same. The network code which i have written is as follows:

Edit

image = Input(shape=(None, None, 3))

# Encoder
l1 = Conv2D(64, (3,3), strides = (2), padding='same', activation='leaky_relu')(image)  
l2 = MaxPooling2D()(l1)
l3 = Conv2D(32, (5,5), strides = (2), padding='same', activation='leaky_relu')(l2)
l4 = MaxPooling2D(padding='same')(l3)
l5 = Conv2D(16, (7,7), strides = (2), padding='same', activation='leaky_relu')(l4)
l6 = MaxPooling2D(padding='same')(l5)
l7 = Conv2D(8, (5, 5), strides = (2), padding = 'same', activation = 'leaky_relu')(l6)
l8 = MaxPooling2D(padding='same')(l7)
l9 = Conv2D(4, (3, 3), strides = (2), padding = 'same', activation = 'leaky_relu')(l8)
l10 = MaxPooling2D(padding='same')(l9)
l11 = Conv2D(2, (4, 4), strides = (2), padding = 'same', activation = 'leaky_relu')(l10)
l12 = MaxPooling2D(padding='same')(l11)
l13 = Conv2D(1, (2, 2), strides = (2), padding = 'same', activation = 'leaky_relu')(l12)

# latent variable z
l14 = Reshape((60,512))(l13)
print(l14.shape)#-->output=(None, 60, 512)
l15 = Dense((512), activation = 'leaky_relu')(l14)
print(l15.shape) #-->output=(None, 60, 512)
l16 = Dense((128), activation = 'leaky_relu')(l15)
print(l16.shape)#-->output=(60, 128)
l17 = Reshape((60,128))(l16)
print(l17.shape) #-->output=(60, 128)

#Decoder
l18 = UpSampling3D(size = (3,3,3))(l17) #-->throws error->IndexError: list index out of range
l19 = Conv3DTranspose(60, (8, 8, 8), strides = (64), padding='same', activation = 'leaky_relu') (l17)
l20 = UpSampling3D((3,3,3))(l19)
l21 = Conv3DTranspose(60, (16,16,16), strides =(32), padding='same', activation = 'leaky_relu')(l20)
l22 = UpSampling3D((3,3,3))(l21)
l23 = Conv3DTranspose(60, (32, 32, 32), strides = (32), padding='same', activation = 'lealy_relu')(l22)
l24 = UpSampling3D((3,3,3))(l23)
l25 = Conv3DTranspose(60, (64, 64, 64), strides = (24), padding='same', activation = 'leaky_relu')(l24)
l26 = UpSampling3D((3,3,3))(l25)
l27 = Conv3DTranspose(60, (64, 64, 64), strides = (1), padding='same', activation = 'leaky_relu')(l26)

model3D = Model(image, l27)

This is giving me endless errors i solved some initially and seems to get stuck at this one really bad!! the error persists at l17, and says:


---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/tmp/ipykernel_33/907378238.py in 
     27 
     28 #Decoder
---> 29 l18 = UpSampling3D(size = (3,3,3))(l17) #-->throws error->IndexError: list index out of range
     30 l19 = Conv3DTranspose(60, (8, 8, 8), strides = (64), padding='same', activation = 'leaky_relu') (l17)
     31 l20 = UpSampling3D((3,3,3))(l19)

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
    975     if _in_functional_construction_mode(self, inputs, args, kwargs, input_list):
    976       return self._functional_construction_call(inputs, args, kwargs,
--> 977                                                 input_list)
    978 
    979     # Maintains info about the `Layer.call` stack.

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _functional_construction_call(self, inputs, args, kwargs, input_list)
   1113       # Check input assumptions set after layer building, e.g. input shape.
   1114       outputs = self._keras_tensor_symbolic_call(
-> 1115           inputs, input_masks, args, kwargs)
   1116 
   1117       if outputs is None:

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _keras_tensor_symbolic_call(self, inputs, input_masks, args, kwargs)
    846       return tf.nest.map_structure(keras_tensor.KerasTensor, output_signature)
    847     else:
--> 848       return self._infer_output_signature(inputs, args, kwargs, input_masks)
    849 
    850   def _infer_output_signature(self, inputs, args, kwargs, input_masks):

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _infer_output_signature(self, inputs, args, kwargs, input_masks)
    886           self._maybe_build(inputs)
    887           inputs = self._maybe_cast_inputs(inputs)
--> 888           outputs = call_fn(inputs, *args, **kwargs)
    889 
    890         self._handle_activity_regularization(inputs, outputs)

/opt/conda/lib/python3.7/site-packages/keras/layers/convolutional.py in call(self, inputs)
   2720   def call(self, inputs):
   2721     return backend.resize_volumes(
-> 2722         inputs, self.size[0], self.size[1], self.size[2], self.data_format)
   2723 
   2724   def get_config(self):

/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py in wrapper(*args, **kwargs)
    204     """Call target, and fall back on dispatchers if there is a TypeError."""
    205     try:
--> 206       return target(*args, **kwargs)
    207     except (TypeError, ValueError):
    208       # Note: convert_to_eager_tensor currently raises a ValueError, not a

/opt/conda/lib/python3.7/site-packages/keras/backend.py in resize_volumes(x, depth_factor, height_factor, width_factor, data_format)
   3215     output = repeat_elements(x, depth_factor, axis=1)
   3216     output = repeat_elements(output, height_factor, axis=2)
-> 3217     output = repeat_elements(output, width_factor, axis=3)
   3218     return output
   3219   else:

/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py in wrapper(*args, **kwargs)
    204     """Call target, and fall back on dispatchers if there is a TypeError."""
    205     try:
--> 206       return target(*args, **kwargs)
    207     except (TypeError, ValueError):
    208       # Note: convert_to_eager_tensor currently raises a ValueError, not a

/opt/conda/lib/python3.7/site-packages/keras/backend.py in repeat_elements(x, rep, axis)
   3248   x_shape = x.shape.as_list()
   3249   # For static axis
-> 3250   if x_shape[axis] is not None:
   3251     # slices along the repeat axis
   3252     splits = tf.split(value=x,

IndexError: list index out of range```
```

At this point i seem to be directionless, any help would be really appreciated. thanks in advance

user_659156 · Answer

The shape of l16 is:

l16.shape
TensorShape([None, 60, 8192])

and now you want to change the shape [60, 8192] into a shape [4,4,4,128] with the call Reshape((4,4,4,128))(l16). But 60 * 8192 = 491520 and 4 * 4 * 4 * 128 = 8192. So those two shapes are incompatible (491520 != 8192). That's why the error message correctly states:

ValueError: total size of new array must be unchanged, input_shape = [60, 8192], output_shape = [4, 4, 4, 128]```

The total number of cells must be the same before and after a reshape. E.g., you can change a (4,) tensor into a (2,2) tensor, but not e.g. into a (3,2) tensor.

The origin lies with l14, which you give the shape [60, 512]:

l14.shape
TensorShape([None, 60, 512])

Now, when you apply a Dense layer to a 2-dim shape like this, it will be applied to the last dimension, i.e. the first dimension of the shape stays the same. That is why l15 still has the shape [60, 512]:

l15.shape
TensorShape([None, 60, 512])

Similarly, l16 will have a shape [60, 128 * 4 * 4 * 4] = [60, 8192]. Then, this is the input into the line for l17 where Reshape chokes as explained above.

Autoencoder to convert 2d to 3d layers ambiguity

1 Answers