About managing logs of prefect workflow

user_14574507 0
Viewed 597

I am using prefect workflow for business applications.

I have a question about logs stored in postgresql.

If daily logs are kept stored on the postgresql server, the amount of data will be enormous.

Is there a mechanism to rotate this log and write it to a text file?

Also, is it okay to delete the corresponding record after writing the data from the postgresql table to a text file, etc.?

prefect airflow workflow postgresql python
1 Answers

Since you're running an instance of Prefect Server, it would make sense to write a Flow that connects to the postgres container and archives the logs to some sort of cloud storage (GCS, S3, etc) for you. I'd probably think about it like this:

  1. Use the Prefect Client to gather all the IDs of Flow Runs older than a certain date.
  2. Connect to postgres and select the logs from the logs table for logs with those Flow Run IDs
  3. Write those logs to CSV/SQL/Text as preferred
  4. Delete those logs from postgres
user_58352210
Related