Keras ImageDataGenerator not handling symlink files

Viewed 102

I am trying to train a DenseNet121 model on chest X-ray images using tensorflow.keras, and using ImageDataGenerator for augmentation. I have directories of files containing symlinks to the images that I believe is set up in the correct format for ImageDataGenerator:

Train

Normal

Abnormal

Val

Normal

Abnormal

However, when I call model.fit(), it throws FileNotFoundError: [Errno 2] No such file or directory: '.\\Train\\Normal\\00017275_014.png' which is a symlink file. .flow_from_directory(follow_links = True) did not solve the problem. Also, calling os.islink() with that path returns True.

In addition: calling imagedatagenerator returns: Found 84090 images belonging to 2 classes. Found 28030 images belonging to 2 classes.

Any suggestions? Code below:

from tensorflow.keras.applications.densenet import preprocess_input
from tensorflow.keras import Model,layers
from tensorflow.keras.preprocessing.image import ImageDataGenerator

from tensorflow.keras.optimizers import Adam, SGD
from tensorflow.keras.metrics import binary_accuracy
from tensorflow.keras.losses import binary_crossentropy

batch_size = 64

train_datagen = ImageDataGenerator(
    preprocessing_function = preprocess_input,
    brightness_range = [0.75, 1.25],
    horizontal_flip=True,
)

train_generator = train_datagen.flow_from_directory(
    directory = '.\\Train',
    color_mode = 'rgb',
    classes = ['Normal', 'Abnormal'],
    class_mode = 'binary',
    batch_size = batch_size,
    target_size = (224,224),
    follow_links=True,
)

val_datagen = ImageDataGenerator(
    preprocessing_function = preprocess_input,
)

val_generator = val_datagen.flow_from_directory(
    directory = '.\\Val',
    color_mode = 'rgb',
    class_mode = 'binary',
    classes = ['Normal', 'Abnormal'],
    batch_size = batch_size,
    target_size = (224,224),
    follow_links = True,
)

from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping

model_name = "Imagenet DenseNet121 on NIH full dataset 375 locked brightness flip.h5"
callback_checkpoint = [
    EarlyStopping(monitor = 'val_loss', patience = 10, verbose = 1),
    ModelCheckpoint(model_name,
                    verbose = 1,
                    monitor = 'val_loss',
                    save_best_only = True,
                   )
]

model.compile(
    optimizer = Adam(),
    #optimizer = SGD(learning_rate = 0.001, momentum = 0.9, decay = 0.0001),
    loss = 'binary_crossentropy',
    metrics = ['binary_accuracy'],
)

history = model.fit(
        train_generator,
        steps_per_epoch=1250,
        epochs=50,
        validation_data=val_generator,
        validation_steps=437,
        callbacks = [callback_checkpoint],
)

`os.path.islink((os.path.join(os.getcwd(), "Train", "Normal", "00017275_014.png")))

True`

1 Answers

At least for pathlib.Path the combined notation with dot and double backslash is not valid. I guess this is the problem here also. Try using forward slashes. Instead of directory = ".\\Val" try

directory = "./Val"

or simply

directory = "Val"
Related