How to replace values in a column in a fast and memory efficient manner

Viewed 168

I have a dataframe containing 114 Million records with 2 columns named session_id and artifact_id. Both the columns are categorical. I would like to replace the values in the column artifact_id with the values in a dictionary where each value in artifact_id is mapped to another value. Artifact_id is an int column and the values to be replaced are also int values. There are about 322k unique values that are to be replaced.

Here's my sample dataset:

Session_id    artifact_id
    A              234
    A              123
    B              123
    B              678

The contents of the dictionary are as follows:

{'234':'1','123':'2','678':'3'}

I would like the final dataset to look like this:

Session_id    artifact_id
    A              1
    A              2
    B              2
    B              3

I would have thought that the following statement would replace these values:

Sessions['artifact_id'].replace(artifactId2Num, inplace=True)

artifactId2Num is the name of the dictionary. This statement gives me an out of memory error. So I thought about breaking the process up into various pieces to avoid MemoryError using the following code:

count = 0
for idx in xrange(0,len(Sessions),50000):
    count = count + 1
    print(count)
    if (idx+50000) > len(Sessions):
        Sessions[idx:(len(Sessions)-1)]['artifact_id'].replace(artifactId2Num, inplace=True)
    else:
        Sessions[idx:(idx+50000)]['artifact_id'].replace(artifactId2Num, inplace=True)

The above code runs so far without any errors. It has been running for about 10+ hours and hasn't finished yet.

MORE INFO: The original dataframe with 114 Million records fits well in the memory and takes about 4.2 GB. The moment I run the above code with iterations, the memory occupancy suddenly increases to almost 20GB. I'm clearly working on only 50000 records at a time and replacing values from the dictionary. Why would the memory usage increase drastically?

Is there a way to make this code faster? Or is there another way I can achieve the same result?

Any help would be much appreciated.

1 Answers
Related