I have a dataframe containing 114 Million records with 2 columns named session_id and artifact_id. Both the columns are categorical. I would like to replace the values in the column artifact_id with the values in a dictionary where each value in artifact_id is mapped to another value. Artifact_id is an int column and the values to be replaced are also int values. There are about 322k unique values that are to be replaced.
Here's my sample dataset:
Session_id artifact_id
A 234
A 123
B 123
B 678
The contents of the dictionary are as follows:
{'234':'1','123':'2','678':'3'}
I would like the final dataset to look like this:
Session_id artifact_id
A 1
A 2
B 2
B 3
I would have thought that the following statement would replace these values:
Sessions['artifact_id'].replace(artifactId2Num, inplace=True)
artifactId2Num is the name of the dictionary. This statement gives me an out of memory error. So I thought about breaking the process up into various pieces to avoid MemoryError using the following code:
count = 0
for idx in xrange(0,len(Sessions),50000):
count = count + 1
print(count)
if (idx+50000) > len(Sessions):
Sessions[idx:(len(Sessions)-1)]['artifact_id'].replace(artifactId2Num, inplace=True)
else:
Sessions[idx:(idx+50000)]['artifact_id'].replace(artifactId2Num, inplace=True)
The above code runs so far without any errors. It has been running for about 10+ hours and hasn't finished yet.
MORE INFO: The original dataframe with 114 Million records fits well in the memory and takes about 4.2 GB. The moment I run the above code with iterations, the memory occupancy suddenly increases to almost 20GB. I'm clearly working on only 50000 records at a time and replacing values from the dictionary. Why would the memory usage increase drastically?
Is there a way to make this code faster? Or is there another way I can achieve the same result?
Any help would be much appreciated.