.foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package

I have a very simple pyspark app that I'm running on EMR Pyspark 3.0.0. See the project structure below, process.py is what controls the flow of the application and calls code inside the file_processor package.

process.py
file_processor
  config        
    spark.py
  repository        
    s3_repo.py
  structure        
    table_creator.py 

I'm seeing strange behavior when I run a .foreachPartition on a small dataframe.

The command hangs when the .foreachPartition code that is located inside s3_repo.py is called by process.py

When the same .foreachPartition code is moved from s3_repo.py and placed inside the process.py it runs just fine.

What is it about the location of the .foreachPartition code that’s making a difference? Is this a Pyspark bug or am I missing something simple here?

process.py code

from file_processor.structure import table_creator
from file_processor.repository import s3_repo


def process():
    table_creator.create_table()
    s3_repo.save_to_s3()


if __name__ == '__main__':
    process()

spark.py code

from pyspark.sql import SparkSession
spark_session = SparkSession.builder.appName("Test").getOrCreate()

s3_repo.py code

from file_processor.config.spark import spark_session

def save_to_s3():
    spark_session.sql('SELECT * FROM rawFileData').toJSON().foreachPartition(_save_to_s3)

def _save_to_s3(iterator):   
    for record in iterator:
        print(record)

table_creator.py code

from file_processor.config.spark import spark_session
from pyspark.sql import Row


def create_table():
    file_contents = [
        {'line_num': 1, 'contents': 'line 1'},
        {'line_num': 2, 'contents': 'line 2'},
        {'line_num': 3, 'contents': 'line 3'}        
    ]
    spark_session.createDataFrame(Row(**row) for row in file_contents).cache().createOrReplaceTempView("rawFileData")