Write Sensor Data to Disk while Minimizing Total Disk Writes and Total Filesize

I am collecting sensor data for several render nodes every 15 seconds, and I plan to let this data collection run indefinitely. The data isn't particularly high resolution and contains a lot of repeated values, so it makes sense to compress it. The python script collecting the data currently looks like this:

POLL_EVERY = 15.0 # seconds
DATANAME = 'sensor_data.parquet'

columns = ['timestamp', 'machine', 'load', 'freq', 'watts', 'temp', etc (many other columns)]
if not os.path.isfile(DATANAME):
  data = {jj:[] for jj in columns}
  df = pandas.DataFrame.from_dict(data)
else:
  print("Reading parquet...")
  df = pandas.read_parquet(DATANAME)

while True:
  for machine in machines:
      data = get_data(machine)
      newrow = pandas.DataFrame([data], columns=columns)
      df = pandas.concat([df, newrow])
  
  print("Writing parquet...")
  df.to_parquet(DATANAME, compression = 'brotli', index=False)
  
  time.sleep(POLL_EVERY)

My concern is that as the size of the .parquet file grows, writing it every single time to disk as I do in my example above will become slower and slower (as well as waste a lot of of the limited writes on the SSD). Ideally I would simply append the additional data to the file each time so it isn't writing the entire history to the disk over again every time it polls the machines for new information.

The problem is that this approach would require the data to be uncompressed (I assume), and I'm getting a significant file size savings using brotli compression (around 1/20th of the size of an uncompressed csv file).

What is the best approach to take here to minimize unnecessary writes but also not end up with a huge multi gigabyte file after it has been monitoring for several months/years? I would like to keep as much of the history as possible.

I'm thinking of potentially keeping an uncompressed csv file that I append to for a while and then having a separate thread which monitors its filesize and eventually appends it to an existing parquet file that keeps the majority of the history once the csv gets too big. That way it only re-writes the entire history occasionally and still keeps most of the data compressed.

But maybe there's a better way than this.

Any suggestions?