How to develop and test python transforms code locally?

The simplest way that doesn't require you to mock the transforms package, would be to just extract your logic into a pure python with pyspark function that receives dataframes as input and returns the dataframe.

i.e.:

# yourtransform.py
from my_business_logic import magic_super_complex_computation

@transform_df(
   Output("/foo/bar/out_dataset"),
   input1=Input("/foo/bar/input1"),
   input2=Input("/foo/bar/input2"))
def my_transform(input1, input2):
   return magic_super_complex_computation(input1, input2)

You can now import in your test the magic_super_complex_computation and test it just with pyspark.

i.e:

from my_business_logic import magic_super_complex_computation


def test_magic_super_complex_computation(spark_context):
    df1 = load_my_data_as_df(spark_context, "input1")
    df2 = load_my_data_as_df(spark_context, "input2")

    result = magic_super_complex_computation(input1, input2).collect()
    assert len(result) == 123

Do note that this requires you to provide a valid spark context as a fixture in your pytest (or whatever testing framework you are using)