I want to ask an advice what the best way to do my task. I have file server that contains zipped json files with size from 5 mb to ~500 mb. I need to read that files as pandas dataframe. But I don't know what's best way to do this.
- I can load file to local machine using
paramiko, unzip,read_jsonit and delete source zip file (but I think it will take a lot of memory) - Read file in stream, using
with, but I don't really know is this possible and how to read file-like zip object. ( I think it will take a lot of time on big files) - Maybe there are other possible solutions?