StreamingFileSink fails to start if an MPU is missing in S3

user_219449 0
Viewed 80

We are using StreamingFileSink in Flink 1.11 (AWS KDA) to write data from Kafka to S3.

Sometimes, even after a proper stopping of the application it will fail to start with:

com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.

By looking at the code I can see that files are moved from in-progress to pending during a checkpoint: files are synced to S3 as MPU uploads and a _tmp_ objects when upload part is too small.

However, pending files are committed during notifyCheckpointComplete, after the checkpoint is done.

streamingFileSink will fail with the error above when an MPU which it has in state does not exists in S3.

Would the following scenario be possible:

  1. Checkpoint is taken and files are transitioned into pending state.
  2. notifyCheckpointComplete is called and it starts to complete the MPUs
  3. Application is suddenly killed or even just stopped as part of a shutdown.
  4. Checkpointed state still has information about MPUs, but if you try to restore from it it's not going to find them, because they were completed outside of checkpoint and not part of the state.

Would it better to ignore missing MPUs and _tmp_ files? Or make it an option?
This way the above situation would not happen and it would allow to restore from arbitrary checkpoints/savepoints.

apache-flink
1 Answers

The Streaming File Sink in Flink has been superseded by the new File Sink implementation since Flink 1.12. This is using the Unified Sink API for both batch and streaming use cases. I don't think the problem you've listed here would occur when using this implementation.

user_161736690
Related