Using tfrec files in Keras

Question

I feel like this should be simple but cannot for the life of me work it out.

I have this melanoma dataset(https://www.kaggle.com/datasets/cdeotte/melanoma-512x512/code) (in tfrec format) downloaded to my local machine.

import os
import cv2
import numpy as np
import pandas as pd
import albumentations
import tensorflow as tf
from tensorflow import keras

features = {'image': tf.io.FixedLenFeature([], tf.string),
      'image_name': tf.io.FixedLenFeature([], tf.string),
      'patient_id': tf.io.FixedLenFeature([], tf.int64),
      'sex': tf.io.FixedLenFeature([], tf.int64),
      'age_approx': tf.io.FixedLenFeature([], tf.int64),
      'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),
      'diagnosis': tf.io.FixedLenFeature([], tf.int64),
      'target': tf.io.FixedLenFeature([], tf.int64),
      'width': tf.io.FixedLenFeature([], tf.int64),
      'height': tf.io.FixedLenFeature([], tf.int64)}

train_filepaths=tf.io.gfile.glob(path+'/train*.tfrec')
train_filepaths

this lists all the files: ['\Users\adban\Dissertation\Moles\512 rain00-2182.tfrec', '\Users\adban\Dissertation\Moles\512 rain01-2185.tfrec', '\Users\adban\Dissertation\Moles\512 rain02-2193.tfrec', ...]

But I cannot seem to decode them. (Tried 'tf.io.parse_single_example' and 'tf.data.TFRecordDataset' but either get a parse error or an empty array returned.)

user_19313065 · Answer

I figured it out. This will add all images to a list as 3d array.

def _parse_image_function(example_proto):
    return tf.io.parse_single_example(example_proto, features)

def preprocess_image(image):
    image = tf.io.decode_image(image, channels=3)
    return image

path = '/Users/adban/Dissertation/Moles/512'
tfimage_set = []
for filename in os.listdir(path):
#change for 
    train_image_dataset = tf.data.TFRecordDataset(path+'/'+filename)
    train_images = train_image_dataset.map(_parse_image_function)
    for image_feature in train_images:
        image_raw = preprocess_image(image_feature['image'])
        image_raw_np = image_raw.numpy()
        tfimage_set.append(image_raw_np)

Using tfrec files in Keras

1 Answers