How to run Spark unit testing in parallel via pytest (and fixture)?

Selva 0
Viewed 10

I am writing unit testing for a spark application. I am using pytest and I have created a fixture to load the spark session once.

When I run one test at a time, it is passing but when I run all the tests together I am getting unexpected behavior. Then, I realize, spark is not multi-threadable. Any way to fix this? Is running pytest in non-parallel mode is the only solution?

Sample code structure,

@pytest.fixture(scope="session")
def spark() -> SparkSession:
    builder = SparkSession.builder.appName("pandas-on-spark")
    builder = builder.config("spark.sql.execution.arrow.pyspark.enabled", "true") 
    return builder.getOrCreate()

def test1(spark):
   df = spark.createDataFrame(dummy_rows)
   # do some transformaton
   # assert

def test2(spark):
   df = spark.createDataFrame(dummy_rows)
   # do some transformaton
   # assert

def testN(spark):
   df = spark.createDataFrame(dummy_rows)
   # do some transformaton
   # assert

pytest -s .
pytest pyspark apache-spark python
1 Answers

With scope="session", you'd have a single Spark session for all the tests, means all variables, all caches, all transformations etc. If you really need to have each transformation completely separated from each test, you should consider having a new Spark session for each test by changing lower scope into class or function. The whole test would run slower, but your logic will be secured.

pltc0
Related