parsing exception: Expected [START_OBJECT] but foud [START_ARRAY]

user_15136899 0
Viewed 1030

I'm trying to convert my python ETLs to airflow. I have an ETL written in python to copy data from Elastic to MSSQL. I've build a DAG with 3 tasks.

task 1- get the latest date from the table in MSSQL

task 2- generate an elastic query based on that date retrieved from the previous task plus some filters (must not and sould) taken from a different table in MSSQL (less relevant). eventually generating a body like so:

{ "query": {
      "bool": {
         "filter": {
            "range": {
               "@timestamp": { "gt": latest_timestamp }
                     }
               },
               "must_not": [],
               "should": [],
               "minimum_should_match": 1
             }
          }
}

task 3- scroll the elastic index using the body generated in the previous task and write the data to mssql.

My DAG fails on the 3rd task with the error: parsing exception: Expected [START_OBJECT] but foud [START_ARRAY]

I've taken the generated body and ran it on elastic in dev tools and it is working fine. So I have no idea what is the problem and how to debug it. Any ideas?

airflow elasticsearch python
2 Answers

I've found the problem. I use XCOM to pass the body between Task2 and Task3. Apparently something in the XCOM is messing with the body (I don't see it in the UI anyway). When I put the logic of the body (task2) and call the search with the same body without passing it via XCOM everything is working as expected. So beware of using XCOM cause it has side effects apparently.

user_151368990

I'm not a big ELK guy, but I would assume a different format is required there. When you are doing "scroll the elastic index", you most probably use some API that expects one query format, while in dev console another query format is expected.

E.g. this thread: https://discuss.elastic.co/t/unable-to-send-json-data-to-elastic-search/143506/3 Kibana Post Search - Expected [START_OBJECT] but found [VALUE_STRING]

So, check what format is expected by API handle you use to scroll through the data. Or, if still unclear, please share the function you use for scrolling in task 3.

Also,

user_59512280
Related