How to log custom Python application logs in Databricks and move it to Azure

shankar 0
Viewed 52

I have a requirement to develop an application in python. The python application will interact with any database and execute sql statements against it. It can also interact with Databricks instance too and query the tables in databricks.

The requirement is that the python application should be platform independent. So the application is developed in such a way that if it runs on databricks, only then it will trigger the spark specific code with in the application. If it is run on a standalone node, it skips. The python programs interacts with Azure blob storages for accessing some files/folders. The python application is deployed on Standalone Node/Databricks as a Wheel.

The issue here is with custom logging. I have implemented custom logging in the python application. There are two scenarios here based on where the application is being run.

  1. Standalone Node
  2. Databricks Cluster.

If the code is run on Standalone Node, then the custom log is initially getting logged into local OS folder and after the application completes successfully/fails, it is moved to azure blob storage. But for some reason if it fails to move the log file to azure storage, it is still available in the local file system of Standalone Node.

If the same approach is followed on Databricks, if the application fails to upload the log file to blob storage, we cannot recover it as the databricks OS storage is volatile. I tried to write the log to dbfs. But It doesn't allow to append.

Is there a way to get the application logs from databricks? Is there a possibility that the databricks can record my job execution and store the logs? As I mentioned, the python application is deployed as wheel and it contains very limited spark code.

azure-python-sdk azure-databricks databricks python-3.x
1 Answers

You can mount your storage account with Azure Databricks. Create a separate mount path and store your log files in that mount location. If you store the log files in the mount location it will directly save log files in the azure storage account.

Please follow below code for mounting and sample custom log file transfer:

dbutils.fs.mount(    
    source = "wasbs://<container>@<storage_account>.blob.core.windows.net/",
    mount_point = "/mnt/<mount_path>",
    extra_configs = {"fs.azure.account.key.<storage_account>.blob.core.windows.net":"<Access_key>"})

enter image description here

Custom logging in azure Databricks

#Creation of file path

from datetime import datetime
import pytz
curr=datetime.now(pytz.timezone('Asia/Kolkata')).strftime("%Y%m%d_%H%M%S")#create timezone
directory="/mnt/"
logfilename="demo123"+curr+"log"
path=directory+logfilename
print(path)

#File Handler
import logging
logger = logging.getLogger('loghj')
logger.setLevel(logging.INFO)
FileHandler=logging.FileHandler(path,mode='a')
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s: %(message)s',datefmt='%m/%d/%Y %I:%M:%S %p')
FileHandler.setFormatter(formatter)
logger.addHandler(FileHandler)
logger.debug( 'debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical ('critical message')

#create partition

from datetime import datetime
import pytz
parti=datetime.now(pytz.timezone('Asia/Kolkata')).strftime("%Y/%m/%d")
print(parti)  

# Past your mount path in dbfss
dbutils.fs.mv("file:"+path,"dbfs:/mnt/hh/log/"+partition+logfilename)

enter image description here

Output:

enter image description here

BhanunagasaiVamsi0
Related