Azure function does not execute other Python functions

Viewed 513

I have the following code in main.py, which is a simplified version of my project where I retrieve some data, clean it and export it to a SQL database in Azure.

The problem is that when I run it locally with Azure functions everything works okay and I can see the logging in my terminal.

But when I run it with azure start --verbose, the function starts correctly, but the execution my function export_data does not work.

Simplified code:

# import packages
# load_dotenv()

def get_data():
    # function which retrieves data


def export_data():
    # function which exports data and uses environment variables


def main(timer: func.TimerRequest) -> None:
    utc_timestamp = datetime.datetime.utcnow().replace(
        tzinfo=datetime.timezone.utc).isoformat()

    if timer.past_due:
        logging.info('The timer is past due! Writing data to database')
        export_data()

    logging.info('Python timer trigger function ran at %s', utc_timestamp)


if __name__ == "__main__":
    export_data()

My code:

import os
import datetime
import logging
import pandas as pd
from sqlalchemy import create_engine
from dotenv import load_dotenv
import azure.functions as func
load_dotenv()


logging.basicConfig(
    format="%(asctime)s.%(msecs)03d [%(levelname)-5s] [%(name)s] - %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
    level=logging.INFO,
)


def get_data():
    # get data
    logging.info("retrieving data")
    url = "https://data.rivm.nl/covid-19/COVID-19_aantallen_gemeente_per_dag.csv"
    df = pd.read_csv(url, sep=";")

    return df


def export_data():
    # export data
    df = get_data()

    uid = os.environ.get("username")
    password = os.environ.get("pw")
    server = '***.database.windows.net'
    database = os.environ.get("database")
    driver = "ODBC Driver 17 for SQL Server"

    connection_string = (
        f"mssql+pyodbc://{uid}:{password}@" f"{server}:1433/{database}?driver={driver}"
    )

    engine = create_engine(connection_string)
    con = engine.connect()

    logging.info("exporting data")
    df.head(10).to_sql('covid_19', con=con, if_exists='replace')


def main(timer: func.TimerRequest) -> None:
    utc_timestamp = datetime.datetime.utcnow().replace(
        tzinfo=datetime.timezone.utc).isoformat()

    if timer.past_due:
        logging.info('The timer is past due! Writing data to database')
        export_data()

    logging.info('Python timer trigger function ran at %s', utc_timestamp)


if __name__ == "__main__":
    export_data()

function.json:

{
  "scriptFile": "main.py",
  "bindings": [
    {
      "name": "timer",
      "type": "timerTrigger",
      "direction": "in",
      "schedule": "0 */5 * * * *"
    }
  ]
}

Edit

So here's the part of the log which shows the functions runs when I initially evoke the function with func start --verbose. Notice the time, it does not happen on the whole 5 minutes, as my schedule is based on that ("schedule": "0 */5 * * * *"):

[2020-10-23T09:23:43.339] The timer is past due! Writing data to database
[2020-10-23T09:23:43.340] retrieving data
[2020-10-23T09:23:45.323] exporting data
[2020-10-23T09:23:46.182] Python timer trigger function ran at 2020-10-23T09:23:43.332697+00:00

Then after this part, the function will run based on the timer trigger, but nothing happens:

[2020-10-23T09:25:00.009] Executing 'Functions.zyppcovid' (Reason='Timer fired at 2020-10-23T11:25:00.0076610+02:00', Id=***)
[2020-10-23T09:25:00.012] Received FunctionInvocationRequest, request ID: ***, function ID: ***, invocation ID: ***
[2020-10-23T09:25:00.013] Function is sync, request ID: ***,function ID: ***, invocation ID: ***
[2020-10-23T09:25:00.013] Python timer trigger function ran at 2020-10-23T09:25:00.012563+00:00
[2020-10-23T09:25:00.013] Successfully processed FunctionInvocationRequest, request ID: ***, function ID: ***, invocation ID: ***
[2020-10-23T09:25:00.014] Executed 'Functions.zyppcovid' (Succeeded, Id=***, Duration=6ms)
2 Answers

The code already have a main function for timer trigger def main(timer: func.TimerRequest) -> None:. So it can't do another main if __name__ == "__main__":. I test it in my side, if I just create a simple python code with main, it can do export_data() under the main function success.

You should take the export_data() out of the if statement. The if statement will only execute if the timer is past due.

You most likely want to export_date() each time the azure function is run.

Try this instead (notice how the user-defined function is no longer in the if conditional statement and should now run whenever the function is run, not only when the timer is past due):

import os
import datetime
import logging
import pandas as pd
from sqlalchemy import create_engine
from dotenv import load_dotenv
import azure.functions as func
load_dotenv()


logging.basicConfig(
    format="%(asctime)s.%(msecs)03d [%(levelname)-5s] [%(name)s] - %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
    level=logging.INFO,
)


def get_data():
    # get data
    logging.info("retrieving data")
    url = "https://data.rivm.nl/covid-19/COVID-19_aantallen_gemeente_per_dag.csv"
    df = pd.read_csv(url, sep=";")

    return df


def export_data():
    # export data
    df = get_data()

    uid = os.environ.get("username")
    password = os.environ.get("pw")
    server = '***.database.windows.net'
    database = os.environ.get("database")
    driver = "ODBC Driver 17 for SQL Server"

    connection_string = (
        f"mssql+pyodbc://{uid}:{password}@" f"{server}:1433/{database}?driver={driver}"
    )

    engine = create_engine(connection_string)
    con = engine.connect()

    logging.info("exporting data")
    df.head(10).to_sql('covid_19', con=con, if_exists='replace')


def main(timer: func.TimerRequest) -> None:
    utc_timestamp = datetime.datetime.utcnow().replace(
        tzinfo=datetime.timezone.utc).isoformat()
    
    logging.info('Writing data to database')
    export_data()
    logging.info('Writing data to database complete')

    if timer.past_due:
        logging.info('The timer is past due!')
        

    logging.info('Python timer trigger function ran at %s', utc_timestamp)
Related