I try to understand what is the difference between strict_append=False and strict_append=True in @incremental decorator.
I watched the video:
https://www.youtube.com/watch?v=R8LVvy4v7Es and tried different combinations (uploading different files, updating records, etc), but still don't understand what is a case for strict_append=True.
I have never used strict_append before, so I'm answering from reading the code. You should be able to also see the code by ctrl + click on the function (or cmd + click. It contains some pydoc which says:
strict_append (bool, optional): If True, the underlying foundry transaction type will be an APPEND. Note that the write operation may not overwrite any files, even auxiliary ones such as Parquet summary metadata or Hadoop SUCCESS files. Incremental writes for all Foundry formats should support this mode.
It seems it was added on version 1.312.0
This is what I see in the code base:
# The mode for an incremental write is either 'append' or 'modify', according to the `strict_append` parameter
mode = 'replace'
if is_incremental:
mode = 'append' if self._strict_append else 'modify'
Later on the mode is used to instantiate the type of transaction. Given the if condition above, it seems that with strict_append to true, you will always get an append transaction, so you won't replace or modify the current file contents.