NoneType error when I do a for loop in Airflow

Viewed 23

I'm working with Airflow 2.3.3. I created my first dyamic dag with two tasks: the first do a request to an API that returns a list of JSONs with the data I need, and the second transforms this list to a Dataframe type so I can manipulate it with pandas to extract just the useful data (the first is the one that runs multiple times). The problem is that from my IDLE (pycharm) the scripts works perfectly but when I run the dag, I get an issue where the for loop that I implement to iterate the dataframe, returns me "NoneType" values when should be rows. I tried to iterate the JSON as dict type and I get the same result: the for loop returns me NoneType values.

This is the for loop I made to get the rows from the dataframe and then the data I need:

for dic in df_consulta:
    for n in dic:
        if n.lower() in datos:
            datos[n.lower()] = dic.get(n)
    n_fiscal = dic['domicilioFiscal']
    for f in n_fiscal:
        if f.lower() in datos:
            datos[f.lower()] = n_fiscal.get(f)
    d_f.loc[len(d_f.index)] = datos

"dic" is a row that contains a dict -> this, in Airflow, returns me a "NoneType" variable and in consequence:

TypeError: 'NoneType' object is not iterable

"datos" is an empty dict with the same name keys that dic, so I can extract the values

"domicilioFiscal" is another dict inside the "dic"

"d_f" is the final dataframe that I built to collect the data with the respective registers

Below is the complete dag code:

import teradatasql
import pandas as pd
from airflow.decorators import task, dag
from airflow.hooks.base_hook import BaseHook

DEFAULT_ARGS = {'owner': 'airflow',
                'depends_on_past': False,
                'retries': 1,
                'retry_delay': timedelta(minutes=5)}

@dag(dag_id='Actualization_Teradata',
     default_args=DEFAULT_ARGS,
     start_date=datetime(2022, 9, 12),
     catchup=False,
     schedule_interval='@daily',
     tags=['AFIP', 'Teradata', 'Actualizacion'])

def my_dag():

    @task
    def consulta_afip():
        conn = BaseHook.get_connection(f'Teradata SYSDBA')
        con_tera = teradatasql.connect(None,
                                       host=f"111.11.11.11",
                                       user=f"{conn.login}",
                                       password=f"{conn.password}",
                                       column_name='false')
        query = ("SELECT top 3 * FROM TABLE WHERE COLUMN_1 = 'DATA'")
        global tsql
        tsql = con_tera.cursor()
        tsql.execute(query)
        col_headers = [i[0] for i in tsql.description]
        rows = [list(i) for i in tsql.fetchall()]
        data = pd.DataFrame(rows, columns=col_headers)

        cuit_lista = []
        for d in data['CUIT']:
            cuit_lista.append(d)
        results = perform_web_requests(cuit_lista, 8)
        tsql.close()

        return results

    @task
    def update_teradata(consultas):
        df = pd.DataFrame(consultas)
        df_consulta = df['Contribuyente']
        print(df_consulta)

        datos = {
            'idpersona': '',
            'tipopersona': '',
            'estadoclave': '',
            'nombre': '',
            'nombreprovincia': '',
            'localidad': '',
            'codpostal': '',
            'direccion': ''
        }

        d_f = pd.DataFrame(columns=['idpersona', 'tipopersona', 'estadoclave', 'nombre', 'nombreprovincia', 'localidad',
                                    'codpostal', 'direccion'])

        for dic in df_consulta:
            for n in dic:
                if n.lower() in datos:
                    datos[n.lower()] = dic.get(n)
            n_fiscal = dic['domicilioFiscal']
            for f in n_fiscal:
                if f.lower() in datos:
                    datos[f.lower()] = n_fiscal.get(f)
            d_f.loc[len(d_f.index)] = datos

        columnas_teradata = ['CUIT', 'SITUACION_JURIDICA', 'ESTADO', 'NOMBRE_COMPLETO', 'PROVINCIA', 'LOCALIDAD',
                             'CODIGO_POSTAL', 'DIRECCION']
        d_f.columns = columnas_teradata

        return d_f
        

    update_teradata.expand(consultas=consulta_afip())


act_teradata = my_dag()

As I said, the problem remains in the for loop in "update_teradata" function, the first one (consulta_afip) works well.

0 Answers
Related