Overall: Our code gets data from Snowflake (=data warehouse), performs certain transformations on it in Python with Spark, and pushes it back to Snowflake.
When I perform this task on my computer, things go well: #inputrows = #outputrows. When someone performs it on their computer, data gets lost. The data that gets lost is random: every time this person performs the task, both differen rows and a different amount of rows is returned. Also performing the task on an EMR doesn't return the right number of rows. We perform the exact same spark-submit command.
We believe it is not in the 'writing to snowflake' part, as performing counts in the code already gives the wrong number of rows before writing to Snowflake.
Is it somehow possible to loose data when using Spark?