During the processing of pyspark, spark-related error logs are output to the screen. However, even if an error occurs, the process will not stop and will complete to the end. Try~except is also not detected as an error. The content of the completed data is also normal.
Has anyone else encountered this kind of phenomenon? Please let me know the proper way to catch the error in pyspark.