I have a beam pipeline written in python that reads two parquet files: state and updates. The pipeline keeps the current data "state" in the state file, reads a updates file and it will update the state file (possibly adding new rows) with the content of updates. Ideally my pipeline should overwrite the state file so that next time my pipeline runs with new updates I am comparing them with the most recent state.
Here is my issue. My starting file is called state.parquet after my pipeline ends, it won't override the file but it will create a new file called state-00000-of-00001.parquet. This makes me realise that this is probably not a really good idea to do this, because when the file will grow I could have the output file sharded across multiple separate files, and will cause problems.
What would be a better approach to accomplish what I am trying to do?