today i'm using pandas as my main tool of data pre-processing in my project, where i need to do some transformations in the data to ensure they are in a correct format, which my python class expects.
So i heard about TF Tansform and tested it a little, but i didn't see any obvious advantage (obviously i'm referring to data transformation itself, not in a machine learning pipeline).
For example, i made a simple code in TFT to uppercase all values in my dataframe column:
upper = tf.strings.upper(input, encoding='', name=None)
The execution time of this pre processing function is: 17.1880068779
This is, in the other hand, the code that i use to do the exactly same thing in dataframe:
x = dataset['CITY'].str.upper()
The execution time is: 0.0188028812408
So,i'm doing something wrong? I think that if we have dozens of transformations and a dataset if millions of lines, maybe TFT will be better in this comparison, but for a 100k dataframe it seems not so useful.