Error updating options: "expanding series: block: XXXXXXXXXXXXXXX : decode postings: invalid size"

I'm not sure whether this applies to your case, but for my future reference i'll quickly write what solved the issue in my case.

As stated before:

I have prometheus deployed as bitnami helm chart on my on-site k8s cluster. The error happens ever since i try to persist the data (because otherwise the charts disappear if the pod is restarted). We use CIFS as a storage system.

Further when i looked into the logs of the pod i found this

kubectl logs prometheus-prometheus-kube-prometheus-prometheus-0

ts=2022-09-09T11:00:35.345Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYH2WTM9X3TQD2T52RPEV6: invalid magic number 0"
ts=2022-09-09T11:00:54.184Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYHP0HMJZ1KVB608P3TV4E duration=2.838654027s
ts=2022-09-09T11:00:54.317Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYHP0HMJZ1KVB608P3TV4E: invalid magic number 0"
ts=2022-09-09T11:01:29.216Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYJR5DA60D17F3W2VAFZAT duration=2.899332494s
ts=2022-09-09T11:01:29.453Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYJR5DA60D17F3W2VAFZAT: invalid magic number 0"
ts=2022-09-09T11:02:32.354Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYMNTDYWWENPT2KSSNWN3H duration=2.900614585s
ts=2022-09-09T11:02:32.494Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYMNTDYWWENPT2KSSNWN3H: invalid magic number 0"
ts=2022-09-09T11:03:35.407Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYPKCES6PDQYB2MPCWFG44 duration=2.913101338s
ts=2022-09-09T11:03:35.554Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYPKCES6PDQYB2MPCWFG44: invalid magic number 0"
ts=2022-09-09T11:04:38.999Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYRGZ3TPP8Q03X7CKVSWHQ duration=3.444085613s
ts=2022-09-09T11:04:39.314Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYRGZ3TPP8Q03X7CKVSWHQ: read symbols: invalid checksum"
ts=2022-09-09T11:05:42.805Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYTF7KEBV6SPBYPR5SE8V3 duration=3.490813666s
ts=2022-09-09T11:05:42.992Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYTF7KEBV6SPBYPR5SE8V3: invalid magic number 0"
ts=2022-09-09T11:06:46.245Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYWDDHPSMCAZ4XK34DWH7D duration=3.252076412s
ts=2022-09-09T11:06:46.392Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYWDDHPSMCAZ4XK34DWH7D: invalid magic number 0"
ts=2022-09-09T11:07:49.278Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYYBASWE0EC9DTPQD72H98 duration=2.884889018s
ts=2022-09-09T11:07:49.435Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYYBASWE0EC9DTPQD72H98: invalid magic number 0"
ts=2022-09-09T11:08:52.155Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGZ08WWSMSG6PBZATX6HT1S duration=2.719327278s
ts=2022-09-09T11:08:52.269Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGZ08WWSMSG6PBZATX6HT1S: invalid magic number 0"
ts=2022-09-09T11:09:55.542Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGZ268E5K4S07EZS7GM9J16 duration=3.271842684s
ts=2022-09-09T11:09:55.893Z caller=head.go:844 level=info component=tsdb msg="Head GC completed" duration=223.829164ms
ts=2022-09-09T11:09:55.912Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=2 mint=1662717600000
ts=2022-09-09T11:10:03.499Z caller=head.go:1013 level=info component=tsdb msg="WAL checkpoint complete" first=0 last=2 duration=7.590667466s

Especially consider: compaction failed; corrupted block XXX invalid magic number 0

For this error case there are far more search results:

• https://github.com/prometheus/prometheus/issues/7397
• https://github.com/prometheus/prometheus/issues/10679

just to name a few. Especially the latter shows that this might well have to do with prometheus not supporting your specific filesystem, which is well possible with CIFS. From previous experience i knew that mounting options are crucial, so i added

mfsymlinks and cache=none to our storageClass. and indeed that resolved the issue.

btw. thanks @Machavity for keeping this thread "clean"