I'm very new to MongoDB and I'm trying to migrate some of the work I do in Python with Pandas to MongoDB. Initially the data was small enough that I could load it using Pandas but this is no longer ideal. I use pymongo to connect to the MongoDB server and run the queries. I have my data in a collection which looks like the table below in CSV format:
Exp,User,Session,Attempt,TimeStep,Category
1T5,0009,220101,0,0,A
1T5,0009,220101,0,1,A
1T5,0009,220101,0,2,A
1T5,0009,220101,0,3,B
1T5,0009,220101,0,4,B
1T5,0009,220101,0,5,A
1T5,0009,220101,1,0,A
1T5,0009,220101,1,1,A
1T5,0009,220101,1,2,B
1T5,0009,220101,1,3,A
1T5,0009,220101,1,4,B
1T5,0009,220101,1,5,B
Or, as a collection:
[
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 0,
'TimeStep': 0,
'Category': 'A'},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 0,
'TimeStep': 1,
'Category': 'A'},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 0,
'TimeStep': 2,
'Category': 'A'},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 0,
'TimeStep': 3,
'Category': 'B'},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 0,
'TimeStep': 4,
'Category': 'B'},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 0,
'TimeStep': 5,
'Category': 'A'},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 1,
'TimeStep': 0,
'Category': 'A'},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 1,
'TimeStep': 1,
'Category': 'A'},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 1,
'TimeStep': 2,
'Category': 'B'},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 1,
'TimeStep': 3,
'Category': 'A'},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 1,
'TimeStep': 4,
'Category': 'B'},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 1,
'TimeStep': 5,
'Category': 'B'}
]
I need to process the data to generate a table of run lengths. This would look like:
Exp,User,Session,Attempt,Category,StartTimeStep,Length
1T5,0009,220101,0,A,0,3
1T5,0009,220101,0,B,3,2
1T5,0009,220101,0,A,5,1
1T5,0009,220101,1,A,0,2
1T5,0009,220101,1,B,2,1
1T5,0009,220101,1,A,3,1
1T5,0009,220101,1,B,4,2
Or, as a MongoDB output:
[
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 0,
'Category': 'A',
'StartTimeStep': 0,
'Length': 3},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 0,
'Category': 'B',
'StartTimeStep': 3,
'Length': 2},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 0,
'Category': 'A',
'StartTimeStep': 5,
'Length': 1},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 1,
'Category': 'A',
'StartTimeStep': 0,
'Length': 2},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 1,
'Category': 'B',
'StartTimeStep': 2,
'Length': 1},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 1,
'Category': 'A',
'StartTimeStep': 3,
'Length': 1},
{'Exp': '1T5',
'User': 9,
'Session': 220101,
'Attempt': 1,
'Category': 'B',
'StartTimeStep': 4,
'Length': 2}
]
I had previously done this using Pandas by grouping the first 4 columns and further grouping each grouping by a function of Category column:
groups = df.groupby(['Exp', 'User', 'Session', 'Attempt'])
for idx, group in groups:
partitions = (group['Category'] != group['Category'].shift()).cumsum()
for _, partition in group.groupby(partitions):
# First row of each partition has the StartTimeStep,Category information and the count is the Length
pass
I'm struggling with computing the output with MongoDB and because of this I'm currently only using it to retrieve the input table data and convert it to a Pandas DataFrame. This is extremely slow as the collection is about 13 gigabytes and it's being transferred over a network. I'm hoping that I can compute the resulting table on the server and only transfer the results which is a considerably smaller table.