Merging CSV files to one MongoDB collection efficiently in Python

Viewed 136

I have data split between 2 CSV files and I'm trying to merge them and store as 1 collection of embedded documents on MongoDB. I manage to do that using Pandas DataFrames, but the groupby step is very slow because of the .apply method.

Is there a faster/better way of doing that? Maybe loading CSV files to MongoDB and then merging the data?

My code using Pandas:

#load CSV
df_inspection = pd.read_csv('inspection.csv')
df_stock = pd.read_csv('stock.csv')

#group Keys
grouped = ( df_stock.groupby(['INSPECTION_KEY'])
                       .apply(lambda x: x[['ITEM_CODE','ITEM_QTY']].to_dict('r'))
                       .reset_index()
                       .rename(columns={0:'ITEMS'}) )

#Merge DFs
df = df_inspection.merge(grouped, on='INSPECTION_KEY', how='left')

#Insert data
j_data = df.to_dict(orient='records')
db.collection.insert_many(j_data)

Subset of CSV files (real files have more than 100k lines) :

inspection.csv

INSPECTION_KEY, INSPECTION_DATE, STORE_ID, STORE_NAME
   1,            20181211,        'A',     'AAA SHOP'
   2,            20160224,        'A',     'AAA SHOP'
   3,            20171013,        'B',     'STORE BB'

stock.csv

INSPECTION_KEY, ITEM_CODE, ITEM_QTY
  1,               001,     5
  1,               002,     7
  2,               006,     2
  2,               002,     3
  3,               005,     8
  3,               001,     2

Desired document scheme:

[
 {
  INSPECTION_KEY,
  INSPECTION_DATE,
  STORE_ID,
  STORE_NAME,
  ITEMS: [
    {ITEM_CODE, ITEM_QTY}
  ]
 }
]

Desired Collection:

[
 {
  INSPECTION_KEY: 1,
  INSPECTION_DATE: 20181211,
  STORE_ID: 'A',
  STORE_NAME: 'AAA SHOP',
  ITEMS: [
    {
      ITEM_CODE: 001, 
      ITEM_QTY: 5
    },
    {
      ITEM_CODE: 002, 
      ITEM_QTY: 7
    }
  ]
 },
 {
  INSPECTION_KEY: 2,
  INSPECTION_DATE: 20160224,
  STORE_ID: 'A',
  STORE_NAME: 'AAA SHOP',
  ITEMS: [
    {
      ITEM_CODE: 006, 
      ITEM_QTY: 2
    },
    {
      ITEM_CODE: 002, 
      ITEM_QTY: 3
    }
  ]
 },
 {
  INSPECTION_KEY: 3,
  INSPECTION_DATE: 20171013,
  STORE_ID: 'B',
  STORE_NAME: 'STORE BB',
  ITEMS: [
    {
      ITEM_CODE: 005, 
      ITEM_QTY: 8
    },
    {
      ITEM_CODE: 001, 
      ITEM_QTY: 2
    }
  ]
 }
]
0 Answers
Related