How to stop SwiftUI change the fixed padding when we switch button content between Text and Image?

So I have two buttons on the bottom of my screen, button A and button B, somewhere along the line I need to replace the image in the button with text, so we do so by changing the Bool in the top.

Although we apply the same modifiers, the padding of button B changes, and the UI moves around, it seems as if the text claims more space. Desired situation: button A and B, should not move around when changing the button Image to Text.

import SwiftUI

private var showImage: Bool = true

struct SwiftUIView: View {
    var body: some View {
    
      VStack {
        
        Spacer()

        Button(action: {
            print("CLICK")
          }) {
            Image(systemName: "a.circle")
              .modifier(TestButtonModifier())
          }
        .padding(10)
        
        Button(action: {
          print("CLICK")
        }) {
          if showImage {
            Image(systemName: "b.circle")
              .modifier(TestButtonModifier())
          } else {
            Text("B")
              .modifier(TestButtonModifier())
          }
        }
      .padding(10)

      } //: VSTACK
    }
}

struct TestButtonModifier: ViewModifier {
  
  func body(content: Content) -> some View {
    content
      .font(.system(size: 52, weight: .regular))
      .frame(minWidth: 0, maxWidth: .infinity, maxHeight: 100)
      .background(Color.black)
      .padding(2)
      .foregroundColor(Color.white)
  }
}

struct SwiftUIView_Previews: PreviewProvider {
    static var previews: some View {
        SwiftUIView()
    }
}