Why should I use the Airflow Hook instead the Python SDK for DataStore?

user_11328323 0
Viewed 114

Some of my Airflow jobs are going to use the Google DataStore. There are at least two obvious possibilities to access DataStore from Airflow:

The usage of the python library is much more convenient in contract to the hook. It has implemented all the nice things one need. The hook is more or less just pure API wrapper.

However, I'm wondering if there are some advantages from using the hook instead of the client library.

airflow google-cloud-platform python
1 Answers

First let me point out that you are referencing to an old version of the Hook. The updated version can be found in provider package here. See this answer for instructions how to install it. Then you can import the hook as:

from airflow.providers.google.cloud.hooks.datastore import DatastoreHook

The updated version might have the functions that were missing in the old version of contrib hook.

The idea of Hooks is to wrap the python library thus saving you a lot of headache. For example: when you use the hook you don't need to handle setting up the connection. The hook does that for you.

You can always use the python library directly however I consider this a bad practice. It's very common to use the same library for different use cases - Hook can be used with more than one operator thus saving you a lot of code duplication.

If the relevant functions from the python library doesn't exist in the Hook you can always create a custom hook - inheriting from the upstream (open source) hook and use it:

from airflow.providers.google.cloud.hooks.datastore import DatastoreHook
Class MyDatastoreHook(DatastoreHook)
      def missing_method(self):
          #wrap a function from the python lib
          conn = self.get_conn()
          function code
user_146244090
Related