How to reshape the (image, label) dataset after py_function

user_1112245 0
Viewed 1606

I am trying to read a custom mapped dataset for training. But after I map the dataset using a py_function, I get unknown shape like:

def process_path(file_path):

    label = get_label(file_path)
    img = tf.io.read_file(file_path)
    img = decode_img(img)

    print('image shape:', img.shape) #this print correctly: image shape: (180, 180, 3)
    print('label shape:', label.shape) #this print correctly: label shape: ()

    return img, label

train_ds = train_ds.map(lambda x: tf.py_function(process_path, [x], (tf.float32, tf.int32))) 
print(train_ds)
# this print unknown shape <PrefetchDataset shapes: (<unknown>, <unknown>), types: (tf.float32, tf.int32)>

This will fail the model.fit(), so I want to reshape the dataset to the correct shape like:

<BatchDataset shapes: ((None, 180, 180, 3), (None,)), types: (tf.float32, tf.int32)>

using:

train_ds = tf.reshape(train_ds, ((None, 180, 180, 3), (None,)))

But this will give an error:

ValueError: Attempt to convert a value (<MapDataset shapes: (<unknown>, <unknown>), types: (tf.float32, tf.int32)>) with an unsupported type (<class 'tensorflow.python.data.ops.dataset_ops.MapDataset'>) to a Tensor.

How can I correctly assign the (image, label) shape in this step?

keras tensorflow python
1 Answers

You don't need py_function here. Let's say you have a folder called /dogs that is full of jpg. You can use these two little functions to load and decode.

The first one returns 1 if the file name (e.g., 'dogs\\dog1.jpg') is in the folder dogs and 0 otherwise.

The second function also takes a file name and transforms it into float between 0 and 1. Then, it also resizes the picture.

Let me know if anything is unclear.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
from glob2 import glob

os.chdir('c:/users/nicol/pictures')

files = glob('*/*jpg')

def get_label(file_path):
    split = tf.strings.split(file_path, sep=os.sep)[0]
    equal = tf.equal(split, 'dogs')
    cast = tf.cast(equal, tf.int32)
    return cast

def process_path(file_path):
    label = get_label(file_path)
    img = tf.io.read_file(file_path)
    img = tf.image.decode_jpeg(img, channels=3)
    img = tf.image.convert_image_dtype(img, tf.float32)
    img = tf.image.resize(img, size=(180, 180))
    return img, label

train_ds = tf.data.Dataset.from_tensor_slices(files).map(process_path)

next(iter(train_ds))

(<tf.Tensor: shape=(180, 180, 3), dtype=float32, numpy=
 array([[[1.41176477e-01, 9.41176564e-02, 1.33333340e-01],
         [1.41176477e-01, 9.41176564e-02, 1.33333340e-01],
         [1.41176477e-01, 9.41176564e-02, 1.33333340e-01],
         ..., 
         [2.63300300e-01, 2.76176542e-01, 4.67582583e-01],
         [2.46176332e-01, 2.59706050e-01, 4.50785339e-01],
         [2.54726082e-01, 2.68909693e-01, 4.59662050e-01]]], dtype=float32)>,
 <tf.Tensor: shape=(), dtype=int32, numpy=1>)

get_label should return an integer, if it's not already the case.

user_109083750
Related