I'm facing some intermittent inconsistency issues with the implicit directories and implicit inferring
Description of my Scenario:
I write a Parquet file into a directory on the bucket and if that parquet ends up empty (Data read from API returned 0 rows) I recursively delete the directory.
During that I'm facing issues with
Precondition not met while deleting gs://bucket_name/1651198141345/000000/ at generation 0. Attempt 1. Retrying:
{"code":412,"errors":[{"domain":"global","location":"If-Match","locationType":"header","message":"At least one of the pre-conditions you specified did not hold.","reason":"conditionNotMet"}],"message":"At least one of the pre-conditions you specified did not hold."}
Since I'm using the HadoopFileSystem I make a check on the delete. If it's false I throw an exception and halt the process. Im trying to understand why I sometimes face this and sometimes don't.
I looked over the source quickly and I think it's failing because the generation id is 0. Generation is 0 if it was an inferred. directory sine I have (the default) infer directory enabled but why is it being inferred? if I was missing an explicit creation of an empty directory wouldn't it have been consistent?
Is the reason behind the failure of the deletion actually the 0 generation id? if so how and why is it happening on an intermittent basis?
I even tried manually debugging my code to see if it was always like this and when I try to debug it it's able to get the GoogleStorageStatusInfo correctly with a proper generation id and not create a shadow implicit directory
Relevant Environment Configs:
Hadoop Connectors: 1.9.17 (Due to an issue with transitive dependencies upgrading is hard but it's hard to get on the next 3 months)
fs.gs.implicit.dir.repair.enable -> false: Due to issues with listStatus returning deleted/not found directories. Disabling auto-repair seemed to fix the issue across many buckets
fs.gs.performance.cache.enable -> false (not set explicitly and working with the default)
Tips are appreciated even if they are a configuration changes or some insights into what is causing this. Thanks!