I have large mbox file which I can parse using the mailbox api and dump to csv
import mailbox
import csv
mbox = mailbox.mbox("emailfile.mbox")
for message in mbox:
with open('mail.csv','w') as fp:
writer = csv.writer(fp,delimiter=',')
for message in mbox:
data = [ (message['Date'],message['From'], message['To'], message['subject']) ]
writer.writerows(data)
print data
How can I do this using pySpark and map reduce? I am not able to figure how to load the file into rdd and use the mailbox api in a map reduce fashion.
Any ideas ?