Why do we need airflow hooks?

user_1443993 0
Viewed 2411

Doc says:

Hooks are interfaces to external platforms and databases like Hive, S3, MySQL, Postgres, HDFS, and Pig. Hooks implement a common interface when possible, and act as a building block for operators. Ref

But why do we need them?

I want to select data from one Postgres DB, and store to another one. Can I use, for example, psycopg2 driver inside python script, which runs by a python operator, or airflow should know for some reason what exactly I'm doing inside script, so, I need to use PostgresHook instead of just psycopg2 driver?

airflow python database
2 Answers

You should use just PostresHook. Instead of using psycopg2 as so:

conn = f'{pass}:{server}@host etc}'
cur = conn.cursor()
cur.execute(query)
data = cur.fetchall()

You can just type:

postgres = PostgresHook('connection_id')
data = postgres.get_pandas_df(query)

Which can also make use of encryption of connections.

So using hooks is cleaner, safer and easier.

user_96211720

While it is possible to just hardcode the connections in your script and run it, the power of hooks will allow to edit environment variables from within the UI.

Have a look at "Automate AWS Tasks Thanks to Airflow Hooks" to learn a bit more about how to use hooks.

user_63720420
Related