Toolbar items flicker on macOS app built with SwiftUI

I'm implementing a macOS app using latest Xcode (12.3) and SwiftUI.

My problem is that items in toolbars of the app jump around when UI updates based on user selection in navigation view. This is probably an issue in SwiftUI, but I would like to find a workaround for the issue, since it makes the app feel really broken.

Toolbar implementation is simplest possible I could come up with:

Text("Detail")
        .toolbar {
            Spacer()
            Button(action: {}) {
                Image(systemName: "trash")
            }
            Button(action: {}) {
                Image(systemName: "heart")
            }
            Button(action: {}) {
                Image(systemName: "star")
            }
        }

Full source code (~60 lines) can be found from my GitHub repository with the video demonstrating the issue better.

I've tried to set explicit frames for the views to mitigate the issue, but that hasn't helped.