Tensorflow None in Dataset Size

user_6447409 0
Viewed 1100

I am using the dataset API and am getting unexpected None values in my dataset shape.

import tensorflow as tf
import os

def load_seti_dataset(path):
    with open(path) as f:
        dataset_file = f.read().splitlines()
    data = tf.data.Dataset.from_tensor_slices(dataset_file)
    data = data.repeat()
    data = data.shuffle(buffer_size=1000)
    data = data.map(parse_records, num_parallel_calls=4)
    data = data.batch(batch_size=2)
    data = data.prefetch(buffer_size=1)
    return data

def load_image(path):
    image = tf.io.read_file(path)
    image = tf.io.decode_png(image, channels=3)
    image = tf.image.resize(image, [512, 384])
    image = 1. - image / 127.5
    return image

def parse_records(line):
    image_path, image_label = tf.io.decode_csv(line, ["", 0])
    image = load_image(image_path)
    return image, image_label

data = load_seti_dataset("test_data.csv")
print(data)

test_data.csv is just a csv with image path and label. Images I am reading are 512x384 with 3 channels. Each label is just a scalar value.

Printing data outputs these dimensions:

<PrefetchDataset shapes: ((None, 512, 384, 3), (None,)), types: (tf.float32, tf.int32)>

What I would expect:

<PrefetchDataset shapes: ((512, 384, 3), (1,)), types: (tf.float32, tf.int32)>

Why are there None values in each of the dimensions in the batch? Why isn't the second value in the batch a just scalar?

tensor tensorflow dimensions dataset python
1 Answers

You can solve this problem by using the drop_remainder parameter in your function call.

1.

dataset = tf.data.Dataset.from_tensor_slices([1,2,3,4,5,6,7])
dataset = dataset.apply(tf.data.experimental.map_and_batch(lambda x:x+1,batch_size=2,drop_remainder=True))
dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)
print(dataset)
<PrefetchDataset shapes: (2,), types: tf.int32>
  1. dataset = tf.data.Dataset.from_tensor_slices([1,2,3,4,5,6,7]) dataset = dataset.map(lambda x:x+1) dataset = dataset.batch(batch_size=2,drop_remainder=True) dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE) print(dataset)

NOTE: 'tf.data.experimental.map_and_batch()' IS DEPRECATED. It will be removed in a future version. However I have mentioned here since you can find code using this function

For better understand you can click on this link TensorFlow

user_102770420
Related