I have data split between 2 CSV files and I'm trying to merge them and store as 1 collection of embedded documents on MongoDB. I manage to do that using Pandas DataFrames, but the groupby step is very slow because of the .apply method.
Is there a faster/better way of doing that? Maybe loading CSV files to MongoDB and then merging the data?
My code using Pandas:
#load CSV
df_inspection = pd.read_csv('inspection.csv')
df_stock = pd.read_csv('stock.csv')
#group Keys
grouped = ( df_stock.groupby(['INSPECTION_KEY'])
.apply(lambda x: x[['ITEM_CODE','ITEM_QTY']].to_dict('r'))
.reset_index()
.rename(columns={0:'ITEMS'}) )
#Merge DFs
df = df_inspection.merge(grouped, on='INSPECTION_KEY', how='left')
#Insert data
j_data = df.to_dict(orient='records')
db.collection.insert_many(j_data)
Subset of CSV files (real files have more than 100k lines) :
inspection.csv
INSPECTION_KEY, INSPECTION_DATE, STORE_ID, STORE_NAME
1, 20181211, 'A', 'AAA SHOP'
2, 20160224, 'A', 'AAA SHOP'
3, 20171013, 'B', 'STORE BB'
stock.csv
INSPECTION_KEY, ITEM_CODE, ITEM_QTY
1, 001, 5
1, 002, 7
2, 006, 2
2, 002, 3
3, 005, 8
3, 001, 2
Desired document scheme:
[
{
INSPECTION_KEY,
INSPECTION_DATE,
STORE_ID,
STORE_NAME,
ITEMS: [
{ITEM_CODE, ITEM_QTY}
]
}
]
Desired Collection:
[
{
INSPECTION_KEY: 1,
INSPECTION_DATE: 20181211,
STORE_ID: 'A',
STORE_NAME: 'AAA SHOP',
ITEMS: [
{
ITEM_CODE: 001,
ITEM_QTY: 5
},
{
ITEM_CODE: 002,
ITEM_QTY: 7
}
]
},
{
INSPECTION_KEY: 2,
INSPECTION_DATE: 20160224,
STORE_ID: 'A',
STORE_NAME: 'AAA SHOP',
ITEMS: [
{
ITEM_CODE: 006,
ITEM_QTY: 2
},
{
ITEM_CODE: 002,
ITEM_QTY: 3
}
]
},
{
INSPECTION_KEY: 3,
INSPECTION_DATE: 20171013,
STORE_ID: 'B',
STORE_NAME: 'STORE BB',
ITEMS: [
{
ITEM_CODE: 005,
ITEM_QTY: 8
},
{
ITEM_CODE: 001,
ITEM_QTY: 2
}
]
}
]