Normalization of tensorflow dataset column

user_13801204 0
Viewed 578

I have a tensorflow dataset:

data = tf.data.experimental.make_csv_dataset(r"data.tsv.gz",
                                             field_delim="\t",
                                             compression_type="GZIP",
                                             batch_size=1000, 
                                             label_name="Y", 
                                             num_epochs=1)

and I want to normalize column X1, I tried this way:

columns = []

x1 = tf.feature_column.numeric_column('X1', normalizer_fn=lambda x: (x - xMean) / xStd)
columns.append(x1)

l = tf.keras.layers.DenseFeatures(columns)

nn = tf.keras.Sequential()
nn.add(l)
nn.compile(optimizer="Adam", loss="binary_crossentropy", metrics=[tf.keras.metrics.AUC(name='auc')])
nn.fit(data, epochs=5)

but I don't know how to compute mean and std.. How can I get one column from my dataset and compute mean and std? Or maybe is there some normalization function in tensorflow?

tensorflow2.0 tensorflow python-3.x python
1 Answers

As of now, you can achieve this using some workaround by calculating mean and deviation before and then load CSV data using tf.data.experimental.make_csv_dataset.

Below is the code snippet.

import tensorflow as tf
import pandas as pd
import numpy as np

df = pd.read_csv("/content/titanic.csv")

def normalize(data):
  Numeric_column = ["Age"]
  x = df[Numeric_column]
  MEAN = np.mean(x)
  STD = np.std(x)
  return (data-MEAN)/STD

df["normalized_age"] = df["Age"].apply(normalize)
#save the normalized dataframe to csv file
df.to_csv("normalized_data.csv",sep='\t',encoding='utf-8')

Now load the normalized CSV file using tf.data.experimental.make_csv_dataset

user_142906970
Related