Airflow elasticsearch hook

user_15136899 0
Viewed 1397

I need help with elasticsearch hook on airflow.

I've imported the following:

from airflow.providers.elasticsearch.hooks.elasticsearch import ElasticsearchHook

and run the command to search an index:

elastic_hook = ElasticsearchHook(elasticsearch_conn_id = 'ELK')

elastic_conn = elastic_hook.get_conn()

res = elastic_conn.search(index=es_index,

size = 10000,

from_= 0,

request_timeout=1000,

body = body, scroll='2m', )

But I'm getting an error that "connection object has no attribute search". I've tried running dir command on the hook object and got the following:

['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_generate_insert_sql', '_serialize_cell', '_set_context', 'bulk_dump', 'bulk_load', 'conn_name_attr', 'conn_type', 'connection', 'connector', 'default_conn_name', 'elasticsearch_conn_id', 'get_autocommit', 'get_conn', 'get_connection', 'get_connections', 'get_cursor', 'get_first', 'get_hook', 'get_pandas_df', 'get_records', 'get_sqlalchemy_engine', 'get_uri', 'hook_name', 'insert_rows', 'log', 'run', 'schema', 'set_autocommit', 'supports_autocommit']

Yet I have no idea how to transform my search command (that follows a scroll right after) with one of this functions.

Any ideas?

airflow elasticsearch hook python
1 Answers

For apache-airflow-providers-elasticsearch>=4.1.0 :

There is ElasticsearchPythonHook which expose the search endpoint from Elasticsearch Python SDK. It was added by PR Example usage:

def use_elasticsearch_hook():
    es_hook = ElasticsearchPythonHook(hosts=["http://localhost:9200"])
    query = {"query": {"match_all": {}}}
    result = es_hook.search(query=query)
    print(result)
    return True

For apache-airflow-providers-elasticsearch<4.1.0 :

There is no search function in the hook. You can only use SQL to interact with Elasticsearch. The hook has get_conn & get_uri functions. Since the hook inherits from DbApiHook all of its functions are also available. You can use the tests as example how to use it. Alternatively you can install the elasticsearch SDK and interact with it directly by creating a custom hook.

user_146244090
Related