Spark to analyse large mbox Mailbox file using map reduce in Python

Viewed 833

I have large mbox file which I can parse using the mailbox api and dump to csv

import mailbox
import csv
mbox = mailbox.mbox("emailfile.mbox")
for message in mbox:
with open('mail.csv','w') as fp:
    writer = csv.writer(fp,delimiter=',')
    for message in mbox:
        data = [ (message['Date'],message['From'], message['To'], message['subject']) ]
        writer.writerows(data)
        print data

How can I do this using pySpark and map reduce? I am not able to figure how to load the file into rdd and use the mailbox api in a map reduce fashion.

Any ideas ?

1 Answers
Related