Unit test for spark read/write function to azure data lake mount path

Viewed 60

I am very new to unit testing and I need to write a unit test for spark read/write function.

I have a function as below for example:

def read_file(path, format, logger):
   df = spark.read.load(path, format = format)
   etc.....
   return df

note : the path is mounted.

now I need to write unit test for this function / mock adls location.

I have checked below link which is using patch to mock blobServiceClient but I am not sure whether this is useful to me as it is using open function of python.

Trying to use patch to mock BlobServiceClient but the real class is called instead

Is it even possible to mock adls and if so how?

Can anyone please help me in this. Any help would be appreciated.

1 Answers

Make sure HDFS is available locally. If you are running your tests on Windows make sure you read this readme https://github.com/steveloughran/winutils

This repo makes a reference to this other repo for newer versions. These 2 github contributors are part of the Hadoop team and these are the binaries I use for my local environment.

Once this setup is complete you can pass local paths. They can be local linux or windows paths.

Also read this article to correctly set PATH items for hadoop binaries: https://sparkbyexamples.com/spark/spark-hadoop-exception-in-thread-main-java-lang-unsatisfiedlinkerror-org-apache-hadoop-io-nativeio-nativeiowindows-access0ljava-lang-stringiz/?msclkid=4333eb3bc05d11ecb99904c712581198

Related