Python script to read all json files from a folder in the airflow project

azaveri7 0
Viewed 23

We have a requirement where we need to run all the json payloads in different files in a directory in our project (airflow).

Each json payload exists in a separate json file in a directory structure json/payloads.

I am trying to write a python function to read all json payloads and send REST calls with payload from these files.

def read_payloads(category):
    payload_dict = {}
    directory = './json' + "/" + 'payload' + "/" + category
    for filename in os.listdir(directory):
        if os.path.isfile(filename):
            f = open(filename)
            payload_dict.update({filename, json.load(f)})
            f.close()


    payload_dict

I am trying to create a dictionary with key as file names and value as json payload.

I need to pass this dictionary to a python function which will use the json payload from these values and make REST calls one by one.

While trying to execute above function, getting error: No such file or directory:

It seems it is due to absolute / relative path issue.

Also how can I extract the json payload from this dictionary using for loop and make REST call with each payload.

Have a small sample code for REST api.

def server_call(url, token, category):
   payload_dict = read_payloads(category)
   
   /*
     code to make REST call for one payload
   */

Please advise on fixing above function and how to iterate the payload dictionary to make REST calls by using json payloads from the values of this dictionary.

airflow python-3.x
1 Answers

Ok I was able to figure this one out.

import json
import os
from pathlib import Path


def read_payload(category):
    category_dict = {}
    directory = Path(__file__).parent.parent / "payloads/" / category
    for filename in os.listdir(directory):
        filepath = Path(directory) / filename
        if os.path.isfile(filepath):
            f = open(filepath)
            category_dict[filename.replace(".json","")] = json.load(f)
            f.close()
        else:
            print(filepath + " is not a valid file.")

    return category_dict



def server_call(BACKEND_URL, token, category):
   url = os.path.join(BACKEND_URL, 'api/v1/category-creation')
   payload_dict = read_payloads(category)
   
   /*
     code to make REST call for one payload
   */
   headers = {
        "token": f'{token}',
        "Content-Type": "application/json",
        "accept": "application/json"
    }

    for category_name, category_payload in payload_dict.items():
        json_payload = json.dumps(payload_payload)
        response = requests.request("POST", url, headers=headers, data=json_payload)
        ##########################
        ## Load as string and parsing
        response_data = json.loads(response.text)
        print(response_data)

        category_id = response_data['id']
        message = 'The category with id: ' + str(category_id) + ' is created successfully. '
        logging.info(message)

    return "Categories created successfully."
azaveri70
Related