Using z transformation in a dataframe

Admin 1
Viewed 530

I'm using RetailRocket as my dataset. I assigned every event a value, view = 1, addtocart =2, transaction = 3. Now I want to use z-transformation to normalize the values. Unfortunately I got an error. Where is my mistake?

This is my z-transformation code:

df = df.sample(frac=1, random_state=42)
x = df[["visitorid", "itemid"]].values
#y = df["code"].values
y = df["code"].apply(lambda x: (x - x.mean()) / x.std()).values
# Assuming training on 90% of the data and validating on 10%.
train_indices = int(0.9 * df.shape[0])
x_train, x_val, y_train, y_val = (
    x[:train_indices],
    x[train_indices:],
    y[:train_indices],
    y[train_indices:],
)
print(y)

I found this formula for z-transformation with numpy:

X = (X - X.mean()) / X.std()

Error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-2712d78bf2a4> in <module>()
      2 x = df[["visitorid", "itemid"]].values
      3 #y = df["code"].values
----> 4 y = df["code"].apply(lambda x: (x - x.mean()) / x.std()).values
      5 # Assuming training on 90% of the data and validating on 10%.
      6 train_indices = int(0.9 * df.shape[0])

1 frames
pandas/_libs/lib.pyx in pandas._libs.lib.map_infer()

<ipython-input-7-2712d78bf2a4> in <lambda>(x)
      2 x = df[["visitorid", "itemid"]].values
      3 #y = df["code"].values
----> 4 y = df["code"].apply(lambda x: (x - x.mean()) / x.std()).values
      5 # Assuming training on 90% of the data and validating on 10%.
      6 train_indices = int(0.9 * df.shape[0])

AttributeError: 'int' object has no attribute 'mean'
numpy python
2 Answers

May be you need this:

y = (df["code"] - df["code"].mean() / df["code"].std().values

I like this method:(High-Performance, if your dataset has more than 15,000 rows)

df.eval('(code-code.mean())/code.std()')
user_115079100

Since you use apply(lambda x: ...), x will simply be a single value. When you try to use x.mean() on that single value there will be an error.

Instead what you want to do is to use mean and std on the whole column. Using apply, this can be done as follows:

col = 'code'
df['z_score'] = df[col].apply(lambda x: (x - df[col].mean()) / df[col].std())

However, it is faster without apply:

df['z_score'] = (df[col] - df[col].mean())/df[col].std()
user_75795470
Related