Create your own tfds.load

Question

This is what I have

# Ratings data.
ratings = tfds.load('movie_lens/100k-ratings', split="train")
# Features of all the available movies.
movies = tfds.load('movie_lens/100k-movies', split="train")

# Select the basic features.
ratings = ratings.map(lambda x: {
    "movie_title": x["movie_title"],
    "user_id": x["user_id"]
})
movies = movies.map(lambda x: x["movie_title"])

Since I don't want to download MovieLens, but my own data set. I tried reading it through pandas. Unfortunately, a data frame does not have a map(...) method. Is there an option that reads my .csv file and transfers it like tfds.load(...)

This is what I tried

# Ratings data.
ratings = pd.read_csv('/content/drive/My Drive/Dataset/Test/ratings.csv')
movies = pd.read_csv('/content/drive/My Drive/Dataset/Test/movies.csv')

The Error

AttributeError: 'DataFrame' object has no attribute 'map'

user_10908375 · Answer

You don't need your own tfds.load, it simply returns a tf.data.Dataset object, which you can easily build yourself. E.g.:

import pandas as pd
import tensorflow as tf

df = pd.read_csv('https://raw.githubusercontent.com/mwas'
                 'kom/seaborn-data/master/iris.csv')

ds = tf.data.Dataset.from_tensor_slices(dict(df)).\
    map(lambda x: x['sepal_width']).\
    batch(4)

next(iter(ds))

Coming from the docs:

Note: Do not confuse TFDS (this library) with tf.data (TensorFlow API to build efficient data pipelines). TFDS is a high level wrapper around tf.data. If you're not familiar with this API, we encourage you to read the official tf.data guide first.

Read more about tf.data.Dataset.

Here's how you can use it in a learning task:

import pandas as pd
import tensorflow as tf

df = pd.read_csv('https://raw.githubusercontent.com/mwas'
                 'kom/seaborn-data/master/iris.csv')

labels = ['versicolor', 'setosa', 'virginica']

ds = tf.data.Dataset.from_tensor_slices((
    df.drop('species', axis=1).values, df['species'].values)).\
    shuffle(150).\
    map(lambda x, y: (x, tf.where(tf.equal(labels, y))[0])).\
    batch(4)

model = tf.keras.Sequential([
    tf.keras.layers.Dense(8, activation='relu'),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(3, activation='softmax')])

model.compile(loss='sparse_categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

history = model.fit(ds, epochs=50)

user_14290697 · Answer

You are trying to use the map method on pandas.Dataframe object, but the map is applied to pandas.Series object.

You can convert your dataframe to tf.data.Dataset object and then you can apply the map() function like tfds dataset.

Below is the sample example with sample data to convert from dataframe object to tf.data object.

df: pd.DataFrame = pd.DataFrame(
    data={
        'feature1': np.random.rand(10),
        'feature2': np.random.rand(10),
        'feature3': np.random.rand(10),
        'target': np.random.randint(0, 3, 10)
    }
)
features = ['feature1', 'feature2', 'feature3'] 

dataset = (
    tf.data.Dataset.from_tensor_slices(
        (
            tf.cast(df[features].values, tf.float32),
            tf.cast(df['target'].values, tf.int32)
        )
    )
)

Create your own tfds.load

2 Answers