How to achieve full-width headers using UICollectionViewCompositionalLayout but still have content inset of items in section?

Simple stuff that became complicated with UICollectionViewCompositionalLayout I want to have a 15px padding on the start and end (left/right) of my sections, so I set contentInset but that causes headers to also have the inset and that looks horrible as cells are visible under both edges of the header.

If I set supplementariesFollowContentInsets to false it gets even worse, as suddenly headers are all over the place. First header slides out of the screen while the rest look good

Below is the code I use for creating sections and headers.

private func preparePlaylistSection(containerSize: CGSize) -> NSCollectionLayoutSection {
    let itemSize = NSCollectionLayoutSize(widthDimension: .fractionalWidth(1), heightDimension: .fractionalHeight(1))
    let item = NSCollectionLayoutItem(layoutSize: itemSize)
    
    // Size of the group
    let fractionalWidth: CGFloat = 0.42
    let widthHeightAspectRatio: CGFloat = 1.28
    let groupSize = NSCollectionLayoutSize(widthDimension: .fractionalWidth(fractionalWidth), heightDimension: .fractionalWidth(fractionalWidth * widthHeightAspectRatio))
    let group = NSCollectionLayoutGroup.horizontal(layoutSize: groupSize, subitem: item, count: 1)

    return prepareSection(group: group)
}

private func prepareSupplementaryViews() -> [NSCollectionLayoutBoundarySupplementaryItem] {
    // Configure header
    let headerSize = NSCollectionLayoutSize(widthDimension: .fractionalWidth(1), heightDimension: .absolute(HeaderContentView.estimatedHeight))
    let header =  NSCollectionLayoutBoundarySupplementaryItem(layoutSize: headerSize, elementKind: CollectionHeaderView.kind.supplementaryViewKind, alignment: .topLeading)
    header.pinToVisibleBounds = true
    header.zIndex = 10

    return [header]
}

private func prepareSection(group: NSCollectionLayoutGroup) -> NSCollectionLayoutSection {
    let layoutSection = NSCollectionLayoutSection(group: group)
    layoutSection.orthogonalScrollingBehavior = .continuous
    // Spacing between items in the group
    layoutSection.interGroupSpacing = 8
    // Content inset of cells
    layoutSection.contentInsets = NSDirectionalEdgeInsets(top: 0, leading: 15, bottom: 0, trailing: 15)
    
    layoutSection.boundarySupplementaryItems = prepareSupplementaryViews()
    // False sets supplementary views to a full width but breaks label position for some reason
    layoutSection.supplementariesFollowContentInsets = false // Here lies the problem
    
    return layoutSection
}

All sections are created the same way, just with different sizes (first section is the largest, second and third are the same)