Restoring annotations from a PDF corrupted by Mac's Preview application from raw PDF text

Question

Summary

The 'Preview' Application for viewing and editing PDFs, native to MacOS has an annoyin bug which seems to exist since 'High Sierra' and is still not resolved in the current OS version 'Catalina'. It manifests in the following way:

Multiple PDFs are opened and annotated with several text-highlights or comments
The PDFs are saved but left open
The Preview application quits unexpectedly but offers to reopen all files
When doing so the files contain a seemingly random subset of all saved annotations
Even when opening the corrupted PDF with Acrobat Reader DC most of the annotations are gone.

Things I tried

Yesterday this bug occurred to me after some significant work on multiple PDF files, causing me to look into the issue. As the files I had open were stored inside a synced folder, I had access to several versions of the files I was working on. I noticed two things:

for some reason, even versions that were synced at a time when I had made most annotations but before the crash occurred would only display a subset of the annotations.
the file sizes were increasing with time, making me assume that the missing annotations were still stored in the file but were somehow corrupted and thus not displayed.

This made me think that Preview saved new annotations in some corrupted form, but is still somehow able to handle viewing them - at least for a while until it crashes.

Searching the web for solutions I came across this thread where qpdf was suggested to 'repair' PDFs (qpdf --qdf original.pdf unpacked.pdf) but this didn't fix the problem in my case.

Instead I decided to look at the source of the raw PDF in text format, to see if I could discover something. As I expected, searching a corrupted PDF for my user name brought up a large list of matches, apparently one for each annotations I had made. On the other hand, doing the same for a file 'repaired' with qpdf only brought up matches for the annotations that would actually display in the PDF. I tried to figure out some pattern in the source code of annotations that showed up vs. those that don't, but was not successful so far.

Question

My hope now is that someone can spot what is going wrong in the source code of annotations that do not show up in order to fix this mess. The question I am interested in is:

Is there some way to edit a PDFs source code (preferably automatically) to recover the annotations that have been corrupted by Mac's Preview app?

Exemplary references

Evidently we first need to figure out know what is going wrong. As I am not allowed to upload the PDFs themselves, I provide some excerpts from the source which I believe to be relevant, feel free to ask for more snippets! The exerpts contain references to the obj identifiers of two notes which appear and two that don't.

notes that appear

"1741 0" - (4 to 5 references)

at line 3865

106 0 obj
<< /Annots [ 116 0 R 117 0 R 118 0 R 119 0 R 120 0 R 121 0 R 122 0 R 123 0 R
1740 0 R 1741 0 R 1742 0 R ] /Type /Page /MediaBox [ 0 0 595.245 793.92 ]
/Resources << /Font << /F1 776 0 R /F2 791 0 R /F11 463 0 R /F4 766 0 R /F5
771 0 R /F7 786 0 R /F8 453 0 R >> /Properties << /MC55 108 0 R /MC56 114 0 R
/MC51 110 0 R /MC52 112 0 R >> /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI
] /ExtGState << /GS3 752 0 R /GS7 751 0 R /GS5 462 0 R >> >> /Parent 739 0 R
/B [ 811 0 R 810 0 R ] /Rotate 0 /Contents 107 0 R /StructParents 7 /CropBox
[ 0 0 595.245 793.92 ] >>
endobj

at line 8953

1739 0 obj
[ 116 0 R 117 0 R 118 0 R 119 0 R 120 0 R 121 0 R 122 0 R 123 0 R 1742 0 R
1741 0 R 1740 0 R 1757 0 R 1759 0 R 1760 0 R 1758 0 R 1791 0 R 1792 0 R 1793 0 R
1815 0 R 1823 0 R 1824 0 R 1957 0 R 1958 0 R ]
endobj
1740 0 obj
<< /F 4 /Type /Annot /Rect [ 297.7432 133.1971 369.7432 169.1971 ] /DA (/Helvetica 12 Tf 0 g)
/Border [ 0 0 0 ] /M (D:20200728145314Z00'00') /C [ 0.984314 0.360784 0.537255
] /Subtype /Popup >>
endobj

at line 8963 (directly after previous reference)

1741 0 obj
<< /C [ 0.984314 0.360784 0.537255 ] /Border [ 0 0 0 ] /DA (/Helvetica 12 Tf 0 g)
/F 4 /Subtype /Highlight /Type /Annot /Rect [ 42.816 110.6414 293.7432 129.1971
] /M (D:20200728145314Z00'00') /AP << /N 1750 0 R >> /Contents (So turning off multi-start to get a better comparison!)
/QuadPoints [ 271.8799 129.1971 293.7432 129.1971 271.8799 121.0995 293.7432
121.0995 42.816 118.739 103.5295 118.739 42.816 110.6414 103.5295 110.6414
] /T (Marco Langiu) >>
endobj

at line 10721 (might not be an actual reference)
```
0001031741 00000 n
```

at line 11414:

106 0 obj
<< /Annots [ 116 0 R 117 0 R 118 0 R 119 0 R 120 0 R 121 0 R 122 0 R 123 0 R
1740 0 R 1741 0 R 1742 0 R ] /Type /Page /MediaBox [ 0 0 595.245 793.92 ]
/Resources << /Font << /F1 776 0 R /F2 791 0 R /F11 463 0 R /F4 766 0 R /F5
771 0 R /F7 786 0 R /F8 453 0 R >> /Properties << /MC55 108 0 R /MC56 114 0 R
/MC51 110 0 R /MC52 112 0 R >> /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI
] /ExtGState << /GS3 752 0 R /GS7 751 0 R /GS5 462 0 R >> >> /Parent 739 0 R
/B [ 811 0 R 810 0 R ] /Rotate 0 /Contents 107 0 R /StructParents 7 /CropBox
[ 0 0 595.245 793.92 ] >>
endobj

"1833 0" - (5 references)

at line 3875

124 0 obj
<< /Annots [ 134 0 R 135 0 R 136 0 R 137 0 R 138 0 R 139 0 R 140 0 R 141 0 R
142 0 R 143 0 R 1833 0 R 1834 0 R ] /Type /Page /MediaBox [ 0 0 595.245 793.92
] /Resources << /Font << /F7 786 0 R /F1 776 0 R /F2 791 0 R /F4 766 0 R /F11
463 0 R /F5 771 0 R >> /Properties << /MC68 130 0 R /MC69 132 0 R /MC63 128 0 R
/MC64 126 0 R >> /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ] /ExtGState
<< /GS3 752 0 R /GS7 751 0 R /GS5 462 0 R >> >> /Parent 739 0 R /B [ 809 0 R
808 0 R ] /Rotate 0 /Contents 125 0 R /StructParents 8 /CropBox [ 0 0 595.245
793.92 ] >>
endobj

at line 9475

1832 0 obj
[ 134 0 R 135 0 R 136 0 R 137 0 R 138 0 R 139 0 R 140 0 R 141 0 R 142 0 R
143 0 R 1833 0 R 1834 0 R 1842 0 R 1850 0 R 1851 0 R 1867 0 R 1866 0 R 1868 0 R
]
endobj

at line 9483 (directly after previous reference)

1833 0 obj
<< /C [ 0.984314 0.360784 0.537255 ] /Border [ 0 0 0 ] /DA (/Helvetica 12 Tf 0 g)
/F 4 /Subtype /Highlight /Type /Annot /Rect [ 33.006 264.2444 284.3759 324.65
] /M (D:20200728150458Z00'00') /AP << /N 1835 0 R >> /Contents (Which result is in agreement with 2017?)
/QuadPoints [ 202.8095 324.65 284.2632 324.65 202.8095 316.5524 284.2632 316.5524
33.006 314.183 284.297 314.183 33.006 306.0854 284.297 306.0854 33.006 303.725
284.36 303.725 33.006 295.6274 284.36 295.6274 33.006 293.2671 284.3759 293.2671
33.006 285.1695 284.3759 285.1695 33.006 282.8 283.9609 282.8 33.006 274.7024
283.9609 274.7024 33.006 272.342 190.5361 272.342 33.006 264.2444 190.5361
264.2444 ] /T (Marco Langiu) >>
endobj

at line 11423

endobj
124 0 obj
<< /Annots [ 134 0 R 135 0 R 136 0 R 137 0 R 138 0 R 139 0 R 140 0 R 141 0 R
142 0 R 143 0 R 1833 0 R 1834 0 R 1967 0 R ] /Type /Page /MediaBox [ 0 0 595.245
793.92 ] /Resources << /Font << /F7 786 0 R /F1 776 0 R /F2 791 0 R /F4 766 0 R
/F11 463 0 R /F5 771 0 R >> /Properties << /MC68 130 0 R /MC69 132 0 R /MC63
128 0 R /MC64 126 0 R >> /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ] /ExtGState
<< /GS3 752 0 R /GS7 751 0 R /GS5 462 0 R >> >> /Parent 739 0 R /B [ 809 0 R
808 0 R ] /Rotate 0 /Contents 125 0 R /StructParents 8 /CropBox [ 0 0 595.245
793.92 ] >>
endobj

at line 11508

1966 0 obj
[ 134 0 R 135 0 R 136 0 R 137 0 R 138 0 R 139 0 R 140 0 R 141 0 R 142 0 R
143 0 R 1833 0 R 1834 0 R 1967 0 R ]
endobj

notes that do not appear

"1454 0" - (2 references)

at line 6194

1256 0 obj
[ 57 0 R 58 0 R 59 0 R 60 0 R 61 0 R 62 0 R 63 0 R 64 0 R 65 0 R 66 0 R 67 0 R
68 0 R 69 0 R 70 0 R 71 0 R 72 0 R 73 0 R 74 0 R 75 0 R 76 0 R 77 0 R 1257 0 R
1258 0 R 1273 0 R 1281 0 R 1282 0 R 1297 0 R 1306 0 R 1307 0 R 1309 0 R 1305 0 R
1308 0 R 1346 0 R 1347 0 R 1345 0 R 1369 0 R 1371 0 R 1370 0 R 1405 0 R 1404 0 R
1421 0 R 1420 0 R 1437 0 R 1445 0 R 1454 0 R 1453 0 R 1464 0 R 1462 0 R 1463 0 R
1479 0 R 1488 0 R 1489 0 R 1487 0 R 1511 0 R 1519 0 R 1900 0 R ]
endobj

at line 7316

1454 0 obj
<< /C [ 0.980392 0.803922 0.352941 ] /Border [ 0 0 0 ] /DA (/Helvetica 12 Tf 0 g)
/F 4 /Subtype /Highlight /Type /Annot /Rect [ 371.6818 519.4665 500.5739 527.5641
] /M (D:20200728132511Z00'00') /AP << /N 1455 0 R >> /Contents (why three? What happens with the other ones?)
/QuadPoints [ 371.6818 527.5641 500.5739 527.5641 371.6818 519.4665 500.5739
519.4665 ] /T (Marco Langiu) >>
endobj

"1957 0" - (3 to 4 references)