Is there a way to convert a large Batch Dataset into a numpy array? The Batch Dataset in question consists of 150,000 image tensors

Viewed 29

So I have used the image_dataset_from_directory function from the tensorflow.keras.preprocessing library to load my image data. It loaded it into a batch dataset. I want to convert this into a numpy array.

img_data = image_dataset_from_directory(
    r'D:/NLP_HUH_Dataset/archive/img_resized/',
    labels=None,
    label_mode="int",
    class_names=None,
    color_mode="rgb",
    batch_size=1000,
    image_size=(256, 256),
    shuffle=True,
    seed=None,
    validation_split=None,
    subset=None,
    interpolation="bilinear",
    follow_links=False,
    crop_to_aspect_ratio=False
)

But if I try converting it to a numpy array using np.array(img_data) it doesn't throw an error but instead it just creates a 0-dimensional array. I have tried using a for loop to iterate through each tensor within img_data, converting each tensor to a numpy array, and then appending it to an array, but it takes too long and my kernel crashes before completing it. It crashes after around 10 minutes.

My system specifications are as follows-

  • **Processor: ** Core i5-8300H
  • **GPU: ** Nvidia 1050Ti
  • **RAM: ** 16GB 2666MHz

Please suggest more efficient ways of converting a batch dataset into a numpy array. Thank you in advance!

0-dimensional Array

Batch Dataset is not subscriptable

0 Answers
Related