Downsampling image do not reduce GPU memory required

I am trying to run a code following this project JSNet using the Stanford 3D data set. Unfortunately, this code do not works due to an OOM error (the code in that article uses CUDA, so there is no option for not using the GPU). I have search for this problem on this site, and learned that I need to down-sample the image. However, when I do so, it lead to an unusual problem. Here are my attempts:

1. The control (no change of code): Program failed due to:

   failed to allocate 3.06G (3282521088 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
   

2. Change batch_size to 1: Program failed due to:

   failed to allocate 3.06G (3282521088 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
   

3. Add a down-sampling code to the input of the code:
   Each and every instances of (this is in the file contain the function to prepare the data set for training):

   data = np.loadtxt(input_filename)
   

   Is now replaced with:

   data = np.loadtxt(input_filename)
   data = data[0:-1:DOWNSAMPLE_FACTOR, :]
   

   with DOWNSAMPLE_FACTOR = 10. All else is kept the same (This is just meant to be a placeholder, and can be replaced with more complex filters later on)
   Program failed due to:

   failed to allocate 3.06G (3282521088 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
   

4. As above, but DOWNSAMPLE_FACTOR = 450 (the highest I could get while avoiding the divide by 0 error)
   Program failed due to:

   failed to allocate 3.06G (3282521088 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
   

5. Reduce the amount of training file:
   Program failed due to:

   failed to allocate 3.06G (3282521088 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
   

As a matter of fact, nothing short of actually changing the files in the Stanford 3D data set seem to change the amount of memory allocated, which I do not wish to do as it is both extremely heavy memory-wise, and also because it is not my data set. How is it that down-sampling the input numpy array (this is the moment it is read from the file, so it is not used in the training model yet) still cause CUDA to allocate the exact same amount of memory (3.06 GB, so basically all of my GPU)?