I came up with a slightly different approach: instead of hiding the buttons, move them down one level to the breadcrumb bar:

This is done using the Customize UI extension with the following configuration:
"customizeUI.stylesheet": {
".tabs-and-actions-container": {
"background-color": "inherit",
},
".tabs-and-actions-container .editor-actions": {
"position": "absolute",
"top": "100%",
"right": "0px",
"height": "22px !important",
"z-index": "1",
"background-color": "inherit",
},
".tabs-and-actions-container .editor-actions .action-item": {
"margin-right": "3px !important",
},
".tabs-and-actions-container .editor-actions .action-item a": {
"font-size": "13px",
},
".tabs-and-actions-container .editor-actions .action-item .codicon": {
"width": "13px",
"height": "13px",
},
".tabs-and-actions-container .tab:last-child": {
"margin-right": "0 !important",
},
".title.tabs.show-file-icons": {
"overflow": "unset !important",
},
}
This solution is theme-independent, so it should work for all colour combinations. The background color for the buttons is always the same as the background color of the tab bar. If you use only one static theme, you could hard-code the background-color for the .tabs-and-actions-container .editor-actions selector to the exact color of the breadcrumb bar for a more seamless design. However, this does not work when switching themes.
The only drawback to this solution is that the buttons overflow the rightmost breadcrumb information, but I'm fine with that. At least the tab bar does not resize anymore, while I still have those buttons.