Using Python3 -- Need to generate compressed csv file for streaming output of pandas

I am new to python. Kinldyhelp me to create a zipped csv file directly for the streaming data.

Details : My csv file should get updated each time a function is called. Instead of creating a .csv file i want to create a compressed .csv file for streaming data.

Note : My data is not generated in one go. df.to_csv(..., compression="gzip") does'nt work in my case.

Raising this question again , as my earlier question was marked duplicate and closed.

Trail1 :

def outputfile(result):
    
    try:
        
        table = pd.DataFrame(result)
        table = table.transpose()
        
        headers = ['IP','account', 'fd','instance_id','index','sourcetype','Product' , 'State']
        
        if os.path.isfile(final_IP_List_Program_Output_csv):
            mode = 'a'
            header = 0
            
        else :
            mode ='w'
            header = headers
            
        with open(final_IP_List_Program_Output_csv,mode=mode,newline='',encoding='utf8',errors='replace') as csvfile:
            content = table.to_csv(csvfile,sep=',',encoding='utf-8',index=False,line_terminator="",header=header,quoting=csv.QUOTE_NONE)
            print("Successfully uploaded the csv file.")
            
        with gzip.open(final_IP_List_Program_Output_csv_gz, 'a') as compressed_file:
            compressed_file.write(table.to_csv(content,sep=',',encoding='utf-8',index=False,line_terminator="",header=header,quoting=csv.QUOTE_NONE).encode())
            print("Successfully uploaded the compressed file")
            
    except Exception as err :
        print("Error Occurred during FileCreation. Kindly recheck the code.",err)
        sys.exit()

This function gets called to update each line. Using the above I am first creating a csv file first and then a zipped csv file. But my zipped csv file itself is very large in size than the normal csv file.

Trail 2:

tf = tarfile.open(final_IP_List_Program_Output_csv_gz, mode="w:gz")
table = pd.DataFrame(result)
with open(final_IP_List_Program_Output_csv,mode=mode, newline='', encoding='utf8', errors='replace') as file:
              table.to_csv(file,sep=',',mode=mode,encoding='utf-8',index=False ,header=header,line_terminator="",quoting=csv.QUOTE_NONE)
              tf.add(final_IP_List_Program_Output_csv,arcname=os.path.basename(final_IP_List_Program_Output_csv))
              tf.close()
              print("Successfully uploaded the csv file.")
                

Using this , I am able to add only the first line to tar.gz file , but the original csv file is having more than one line.