I know variables function in Airflow uses internal Metadata DB. Does AirFlow have maximum records limitation for variables function? Is there no limit to the number of variables as long as disk space is available? I could not find specific description in official documents.
1 Answers
There is no limit to the number of variables you can store.
Variables is a table in the MySQL/Postgres/MsSQL database that you use as backend. Tables don't have maximum number of records.
Some tables as they get larger may cause performance issue but even if probably variables table won't be the one to cause such problems.
Should this be a concern for you - you can always use alternative secret backend by using this the connections/variables will not be stored on Airflow database but on other storage at your choice (for example: Google Secret Manager, Vault, etc..) You can see the full list on this doc. You can also roll your own secret backend if you wish.
Related