I had (maybe two years ago) written up a tool to zip up several csv files written out to disk in a code repo from a dataframe for someone who's working in a platform that would work best with a zipped up csv file so they can download it and work with it elsewhere (more user friendly for some).
I can't remember if I had gotten it to work at the time but here's my recent stab at this (and yes before you ask I'm aware that there's an easy way to gzip a file using the df.write_dataframe() option... I'm doing this to have more control over the name of the zip and this is a windows user with a locked down system and a select set of tools) ...
Here's what I've got so far in utils.py:
import tempfile
import zipfile
def zipit(source_df, out, zipfile_name, internal_prefix, fileSuffix):
file_list = list(source_df.filesystem().ls())
fs = source_df.filesystem()
zipf = zipfile.ZipFile("zipfile.zip", 'w', zipfile.ZIP_DEFLATED)
for files in file_list:
temp = tempfile.NamedTemporaryFile(prefix=internal_prefix, suffix=fileSuffix)
with fs.open(files.path, 'rb') as f:
w = open(temp.name, 'wb')
w.write(f.read())
w.close()
f.close()
zipf.write(temp.name)
zipf.close()
with open("zipfile.zip", 'rb') as f:
with out.filesystem().open(zipfile_name, 'wb') as w:
w.write(f.read())
w.close()
f.close()
My issue is that this zips up the .csv file(s) and I can name it but it dumps it in a long crazy series of temp folder names and the .csv file crashes when you try to open it.
I'm sure I can figure this out but I feel like I'm blowing this way out of the water here and would appreciate the community wisdom on this.
The other (much less important) problem is that the file itself has a prefix and a suffix which is nice, but it'd be nicer if I could just name the whole file instead of getting the temp files random chars in the middle of the name.