Geting the error: UnboundLocalError: local variable 'file_format' referenced before assignment

Viewed 20

running into following error with the below code,where is it going wrong? any cleanup suggestions also accepted

Traceback (most recent call last):
  File "/home/nishantup/PrescriberPipeline/src/main/python/bin/run_presc_pipeline.py", line 40, in main
    df_city = load_files(spark = spark, file_dir = file_dir, file_format =file_format , header =header, inferSchema = inferSchema)
UnboundLocalError: local variable 'file_format' referenced before assignment

Following is the code:

This is the main function for analysis. There are many imports but from the error traceback, it seems to be flowing from load_files function which was created in another function given at the end.

run_presc_pipeline.py

### Import all the necessary Modules
import get_all_variables as gav
from create_objects import get_spark_object
from validations import get_curr_date, df_count, df_top10_rec, df_print_schema
import sys
import logging
import logging.config
import os
from presc_run_data_ingest import load_files
from presc_run_data_preprocessing import perform_data_clean
from presc_run_data_transform import city_report, top_5_Prescribers
from subprocess import Popen, PIPE

### Load the Logging Configuration File
logging.config.fileConfig(fname='../util/logging_to_file.conf')
from presc_run_data_transform import city_report

def main():
    try:
        logging.info("main() is started ...")
        ### Get Spark Object
        spark = get_spark_object(gav.envn,gav.appName)
        # Validate Spark Object
        get_curr_date(spark)

        ### Initiate presc_run_data_ingest Script
        # Load the City File
        file_dir="PrescPipeline/staging/dimension_city"
        proc = Popen(['hdfs', 'dfs', '-ls', '-C', file_dir], stdout=PIPE, stderr=PIPE)
        (out, err) = proc.communicate()
        if 'parquet' in out.decode():
           file_format = 'parquet'
           header='NA'
           inferSchema='NA'
        elif 'csv' in out.decode():
           file_format = 'csv'
           header=gav.header
           inferSchema=gav.inferSchema

        df_city = load_files(spark = spark, file_dir = file_dir, file_format =file_format , header =header, inferSchema = inferSchema)

        # Load the Prescriber Fact File
        file_dir="PrescPipeline/staging/fact"
        proc = Popen(['hdfs', 'dfs', '-ls', '-C', file_dir], stdout=PIPE, stderr=PIPE)
        (out, err) = proc.communicate()
        if 'parquet' in out.decode():
           file_format = 'parquet'
           header='NA'
           inferSchema='NA'
        elif 'csv' in out.decode():
           file_format = 'csv'
           header=gav.header
           inferSchema=gav.inferSchema

        df_fact = load_files(spark = spark, file_dir = file_dir, file_format =file_format , header =header, inferSchema = inferSchema)

        ### Validate run_data_ingest script for city Dimension & Prescriber Fact dataframe
        df_count(df_city,'df_city')
        df_top10_rec(df_city,'df_city')

        df_count(df_fact,'df_fact')
        df_top10_rec(df_fact,'df_fact')

        ### Initiate presc_run_data_preprocessing Script
        ## Perform data Cleaning Operations for df_city and df_fact
        df_city_sel,df_fact_sel = perform_data_clean(df_city,df_fact)

        #Validation for df_city and df_fact
        df_top10_rec(df_city_sel,'df_city_sel')
        df_top10_rec(df_fact_sel,'df_fact_sel')
        df_print_schema(df_fact_sel,'df_fact_sel')

        ### Initiate presc_run_data_transform Script
        df_city_final = city_report(df_city_sel,df_fact_sel)
        df_presc_final = top_5_Prescribers(df_fact_sel)

        #Validation for df_city_final
        df_top10_rec(df_city_final,'df_city_final')
        df_print_schema(df_city_final,'df_city_final')
        df_top10_rec(df_presc_final,'df_presc_final')
        df_print_schema(df_presc_final,'df_presc_final')

        # Set up Logging Configuration Mechanism

        ### Initiate run_data_extraction Script
        # Validate
        # Set up Error Handling
        # Set up Logging Configuration Mechanism

        ### End of Application Part 1
        logging.info("presc_run_pipeline.py is Completed.")

    except Exception as exp:
        logging.error("Error Occured in the main() method. Please check the Stack Trace to go to the respective module "
              "and fix it." +str(exp),exc_info=True)
        sys.exit(1)

if __name__ == "__main__" :
    logging.info("run_presc_pipeline is Started ...")
    main()

The load_files function comes from the below function:

presc_run_data_ingest.py

import logging
import logging.config

# Load the Logging Configuration File
logging.config.fileConfig(fname='../util/logging_to_file.conf')

# Get the custom Logger from Configuration File
logger = logging.getLogger(__name__)

def load_files(spark, file_dir, file_format, header, inferSchema):
    try:
        logger.info("load_files() is Started ...")
        if file_format == 'parquet' :
            df = spark. \
                read. \
                format(file_format). \
                load(file_dir)
        elif file_format == 'csv' :
            df = spark. \
                read. \
                format(file_format). \
                options(header=header). \
                options(inferSchema=inferSchema). \
                load(file_dir)
    except Exception as exp:
        logger.error("Error in the method - load_files(). Please check the Stack Trace. " + str(exp))
        raise
    else:
        logger.info(f"The input File {file_dir} is loaded to the data frame. The load_files() Function is completed.")
    return df

I did refer to similar qs submitted here but it did not help much:

Similar qs on SO

Any help would be appreciated..

1 Answers

Issue: You have an if..elif in the main function of your run_presc_pipeline.py file. As the file_format variable is only set inside the if...elif, it is possible that if neither the conditions for the if or the elif are True then your file_format variable will not be set.

Solution: You can resolve this in several ways. For example, you could add an else part that sets file_format variable to some default value. E.g. setting it to a text file:

if 'parquet' in out.decode():
    # your code
elif 'csv' in out.decode():
    # your code
else:
    file_format = 'txt'

Alternatively, you can set the file_format variable to None before your if..else statement. However, you may need to handle any resulting exception appropriately.

file_format = None
if 'parquet' in out.decode():
    # your code
elif 'csv' in out.decode():
    # your code
Related