Best practice to load json file as pandas dataframe from file server

Viewed 20

I want to ask an advice what the best way to do my task. I have file server that contains zipped json files with size from 5 mb to ~500 mb. I need to read that files as pandas dataframe. But I don't know what's best way to do this.

  1. I can load file to local machine using paramiko, unzip, read_json it and delete source zip file (but I think it will take a lot of memory)
  2. Read file in stream, using with, but I don't really know is this possible and how to read file-like zip object. ( I think it will take a lot of time on big files)
  3. Maybe there are other possible solutions?
0 Answers
Related