Python apply a lambda function into a csv file(Large files)

Viewed 1367

I want to apply a this function hideEmail to a specific column of my csv file (large file) using python

Example of function :

def hideEmail(email):
    #hide email
    text = re.sub(r'[^@.]', 'x', email)
    return text 

Csv file (large file > 1gb):

    id;Name;firstName;email;profession
    100;toto;tata;test@test.com;developer
    101;titi;tete;test@test.com;doctor
    ..
    ..

5 Answers

Load the csv data into a DataFrame:

df = pd.read_csv(r'/path/to/csv')

Then you can just use pd.Series.str.replace directly as it supports regex by default:

df = df.astype(str).apply(lambda x: x.str.replace(r'[^@.]', 'x'), axis=1)

That said, if all you want to do is changing a large csv file, pandas is probably an overkill.. You might have a look at sed. Here's one example:

sed -E 's/(\w+)@(\w+)/xxx@xxx/' /path/to/file.csv > /path/to/new_file.csv

Its a bit hard to know without the data frame, but you can try:

import pandas as pd #import pandas
df = pd.read_csv('enter_file_path_here') #read the data

df['col'] = df['col'].apply(lambda x: hideEmail(x))
#if you want to make it back to a csv:
df.to_csv('name.csv')

Using pandas

You can use pandas as described here in a previous question to apply a function passed as parameter.

To export the dataframe obtained, use to_csv function described here

import pandas as pd

def hideEmail(email):
    #hide email
    text = re.sub(r'[^@.]', 'x', email)
    return text 
    

column_name = "email"

df = pd.read_csv(r'Path of your CSV file\File Name.csv')
df[column_name] = df[column_name].map(hideEmail)
df.to_csv(r'Path where you want to store the exported CSV file\File Name.csv')

You can use built-in map() function to get it done as follows:

def hideEmail(email):
    #hide email
    text = re.sub(r'[^@.]', 'x', email)
    return text


with open('path/to/csvfile', 'r') as file:
     lines = [l.strip().split(';') for l in file.readlines()]

modifiedlines = []       # to store lines after email field is modified 

for i in lines[1:]:         # iterating from index 1 as index 0 is header
    i[3] = hideEmail(i[3])       # as email field is at index 3
    modifiedlines.append(';'.join(i))     # appending modified line

with open('path/to/csvfile', 'w') as file:
     file.writelines(modifiedlines)            # writing the lines back to file

You can use the built-in map() method to map the function to each line of the file:

import re

def hideEmail(email):
    #hide email
    text = re.sub(r'[^@.]', 'x', email)
    return text 

with open('file.csv', 'r') as r:
    r = map(hideEmail, r.readlines())

with open('file2.csv', 'w') as f:
    for line in r:
        f.write(line + '\n')

EDIT (credits to juanpa.arrivillaga for pointing it out):

The r = map(hideEmail, r.readlines()) can be replaced with just r = map(hideEmail, r).

Related