How to load custom data into tfds for keras cyclegan example?

Question

As per the example in https://keras.io/examples/generative/cyclegan/, a pre-existing dataset has been loaded for implementation. I am trying to add my dataset.

import tensorflow_datasets as tfds
data = tfds.folder_dataset.ImageFolder('Images', shape=(256, 256, 3))
ds = data.as_dataset()

where 'Images' is the root folder containing two subfolders train and test. train folder containing trainA and trainB , test containing testA and testB.

However, I am unable to understand on how to access trainA , trainB , testA and testB so that it gets accepted by keras cyclegan example.

user_5763590 · Answer

Best practice is to write your own tensorflow dataset

you can do so with the TFDS CLI (command line interface).

Install the TFDS CLI: pip install -q tfds-nightly
Navigate into the directory of your dataset: cd path/to/my/project/datasets/
Create a new dataset: tfds new my_dataset
[...] Manually modify my_dataset/my_dataset.py to implement your dataset.
Navigate into your new dataset: cd my_dataset/
Build your new TFDS dataset: tfds build

Within your project you then need to import your dataset

import my.project.datasets.my_dataset

and access it as you would any other tfds dataset:

ds = tfds.load('my_dataset')

Tensorflow documentation for adding a dataset is to be found here.

user_3701747 · Answer

I curated/wrote the whole code here

https://github.com/asokraju/Soiled

and added a read me file with specific instructions on how-to. Hope this is helpful

Custom Tensorflow Input Pipeline for Cycle GANs

Steps to create the dataset

Organize the data set inside a Data.zip file
```
trainA
trainB
testA
testB
```
A and B represents the two classes.
Provide the path ( of the Data.zip file ) in line 28 of Soiled.py i.e.,
```
_DL_URLS = Soiled":"C:\Users\\Downloads\Data_001.zip"}
```
cd into Soiled folder and use tfds build command to build the data
The Tensorflow record files can be found at C:\Users\ ensorflow_datasets\soiled. If needed, these files can be taken elsewhere to use.

loading the data

There are multiple ways to do it.

Import the necessary packages:

import tensorflow as tf
import tensorflow_datasets as tfds
import sys

Ensure that the path to Soiled folder containg the code, NOT the data generated, is accessable to the code. For this I have added the path as follows:
```
sys.path.insert(1, 'C:\Users\\Downloads\')
```

Then the data can be loaded using:

ds = tfds.load('Soiled')
ds

{'trainA': ,
'trainB': ,
'testA': ,
'testB': }

test:

next(iter(ds['trainA']))

Output exceeds the size limit. Open the full output data in a text editor
{'image': ,
'label': }

Steps used to create the folder structure.

Install tensorflow_datasets package
On Command line type tfds new Soiled. This will create a Soiled folder with file structure
```
checksums.tsv
dummy_data/
Soiled.py
Soiled_test.py
```
edit Soiled.py as needed.

Possible issues:

If it fails to build the pipeline, delete the folder tesorflow_datasets folder BEFORE you retry. In windows it can found at C\users\.

If it gives an error something similar to

# tensorflow.python.framework.errors_impl.NotFoundError: Could not find directory C:\Users\	ensorflow_datasets\downloads\extracted\ZIP.Users_kkosara_Downloads_Data_18r38_Co4F-G6ka9wRk2wGFbDPqLZu8TekEV7s9L9enI.zip	estA	rainA

try changing the data_dirs in lines to path_to_dataset or something that ensures it has the correct path to the downloaded data.

Ensure that the folder structure is proper

    1. Organize the data set inside a `Data.zip` file 
    trainA
    trainB
    testA
    testB
    A and B represents the two classes.

also ensure that there are nothing else except the image files inside the folder.