What are these `a.bits.user.amba_prot` signals and why are they only uninitialized conditionally in my HarnessBinder?

Context: I began using Chipyard about a month back to facilitate the building of a quick prototype using RISC-V cores on the VCU118. Chipyard was perfect, but required me to step up and learn Chisel and Rocket-chip tools to extend interconnection to my design.

The first piece of hardware to hook up was the PCIe to AXI Bridge IP provided by Xilinx known as XDMA. fpga-shells provides a wrapper, shell, and overlay for this IP already, so with some studying of the Chipyard System, HarnessBinders, and IOBinders, I managed to hook it up. The overlay was placed like so:

val overlayOutput = dp(PCIeOverlayKey).last.place(PCIeDesignInput(wrangler=dutWrangler.node, corePLL=harnessSysPLL)).overlayOutput
    val (pcieNode: TLNode, pcieIntNode: IntOutwardNode) = (overlayOutput.pcieNode, overlayOutput.intNode)
    val (pcieSlaveTLNode: TLIdentityNode, pcieMasterTLNode: TLAsyncSinkNode) = (pcieNode.inward, pcieNode.outward)

There are two slaves, but I'll only show the IOBinders and HarnessBinders for one. I'm assuming my other Port mixin is functioning correctly since it's exactly like the CanHaveMasterTLMMIO port, but with a separate key and different address ranges. I realize this is inefficient, but was easier than creating an external bus. Here is the IOBinder which takes advantage of CanHaveMasterTLMMIOPort to introduce a master MMIO port to the system bus.

class WithXDMASlaveIOPassthrough extends OverrideIOBinder({
    (system: CanHaveMasterTLMMIOPort) => {
        val io_xdma_slave_pins_temp = IO(DataMirror.internal.chiselTypeClone[HeterogeneousBag[TLBundle]](system.mmio_tl)).suggestName("tl_slave_mmio")
        io_xdma_slave_pins_temp <> system.mmio_tl 
        (Seq(io_xdma_slave_pins_temp), Nil)

    }
})

I retrieve and connect the slave nodes in the TestHarness like so:

 val inParamsMMIOPeriph = topDesign match { case td: ChipTop => 
        td.lazySystem match { case lsys: CanHaveMasterTLMMIOPort => 
              lsys.mmioTLNode.edges.in(0)
        }
    }
 val inParamsControl = topDesign match {case td: ChipTop => 
        td.lazySystem match { case lsys: CanHaveMasterTLCtrlPort => 
            lsys.ctrlTLNode.edges.in(0)
        }
    }

 val pcieClient = TLClientNode(Seq(inParamsMMIOPeriph.master))
 val pcieCtrlClient = TLClientNode(Seq(inParamsControl.master))
 val connectorNode = TLIdentityNode()


 // pcieSlaveTLNode should be driven for both the control slave and the axi bridge slave
 connectorNode := pcieClient 
 connectorNode := pcieCtrlClient 
 pcieSlaveTLNode :=* connectorNode

Finally, I connect the node to a harness binder. I followed the way DDR was connected in Chipyard for this step.

class WithPCIeClient extends OverrideHarnessBinder({
    (system: CanHaveMasterTLMMIOPort, th: BaseModule with HasHarnessSignalReferences, ports: Seq[HeterogeneousBag[TLBundle]]) => {
        require(ports.size == 1)
        th match { case vcu118th: FCMVCU118FPGATestHarnessImp => {
            val bundles = vcu118th.fcmOuter.pcieClient.out.map(_._1)
            val pcieClientBundle = Wire(new HeterogeneousBag(bundles.map(_.cloneType)))
            pcieClientBundle <> DontCare // Some signals aren't being driven to this bundle, but it's hard to know how critical that is. Only happens when myPeripheral is on. 
            bundles.zip(pcieClientBundle).foreach{case (bundle, io) => bundle <> io}
            pcieClientBundle <> ports.head 
        } }
    }
})

I added the pcieClientBundle <> DontCare to surpress these undriven signals. This problem only affects the signals driven to both slaves.

Signals such as:

a.bits.user.amba_prot.fetch

a.bits.user.amba_prot.secure

a.bits.user.amba_prot.modifiable

a.bits.user.amba_prot.privileged

are undriven (resulting in a $RefNotInitializedException in FIRRTL pass through) in both wires. I recognize these come from TLToAXI4 but what's weird is that they are only undriven when I connect another peripheral to the bus. This peripheral does not leave ChipTop. It has three master AXI buses connected to the system like so:

( peirpheralMasterNode1         
        := TLBuffer(BufferParams.default)
        := TLWidthWidget(8) 
        := AXI4ToTL()
        := AXI4UserYanker(capMaxFlight=Some(16)) 
        := AXI4Fragmenter()
        := AXI4IdIndexer(idBits=3)
        := AXI4Buffer()
        := peripheralTop.Master1)
fbus.fromPort(Some("PERIPHERAL_MASTER_1"))() := peripheralMasterNode1

The peripheral's AXI Lite slave is where I suspect the problem could be. Its node is declared like so:

 val regCfgSlv = AXI4SlaveNode(Seq(AXI4SlavePortParameters(
    slaves = Seq(AXI4SlaveParameters(
      address       = AddressSet.misaligned(0xf000E0000L, 0x1000L),
      resources     = regCfgDevice.reg("config"),
     // executable    = true, // Determines whether processor can execute from this memory.
      supportsWrite = TransferSizes(1, 4),
      supportsRead  = TransferSizes(1, 4),
      interleavedId = Some(0))),
  beatBytes = 4
  )))

And it is connected to the system bus using toSlave

 sbus.toSlave(Some(portName)){
      (peripheralTop.regCfgSlv 
        := AXI4Buffer() 
        := AXI4UserYanker(capMaxFlight=Some(2))
        := TLToAXI4()
        // := TLWidthWidget(sbus.beatBytes))
        := TLFragmenter(4, 
                        p(CacheBlockBytes),
                       // sbus.beatBytes,
                        holdFirstDeny = true) 
        := TLWidthWidget(sbus.beatBytes))
    }

When I include this mixin peripheral in my config by setting myPeripheral's key, I get the error about a.bits.user.amba_prot signals not being driven for both HarnessBinders Client Bundles [val pcieClientBundle = Wire(new HeterogeneousBag(bundles.map(_.cloneType)))]. When I only use one of myPeripheral or XDMA, the $RefNotInitializedException goes away.

Here are my configs. I have tried moving WithMyPeripheral around to no avail.

class CustomRocketConfig extends Config(
  new WithPCIeMMIOPort ++  // add default external master port
  new WithControlPort ++ // add control port for pcie cfg. // TODO: Crossbar this on MMIO Port? Move both MMIO and Control to a port on System Bus?
  new freechips.rocketchip.subsystem.WithDefaultSlavePort ++ // add default external slave port
  new WithMyPeripheral(MyPeripheralParams()) ++ // Link up myPeripheral
  new freechips.rocketchip.subsystem.WithNBigCores(2) ++
  new freechips.rocketchip.subsystem.WithNExtTopInterrupts(3) ++
  new chipyard.config.AbstractConfig)

class WithPCIeTweaks extends Config (

    new WithPCIeClient ++ 
    new WithPCIeManager ++
    new WithPCIeCtrlClient ++  // Same for these harness binders - ME
    new WithXDMAMasterIOPassthrough ++ 
    new WithXDMASlaveIOPassthrough ++ 
    new WithXDMACtrlIOPassthrough  // I imagine these three IOBinders can be combined into one - ME



    //TODO: Probably need harness binder and io binder for interrupt if we use it
    )

class myRocketConfig extends Config (
    // new WithMyPeripheral(MyPeripheralParams()) ++ // Link up myPeripheral
    new WithPCIeTweaks ++ 
    new WithVCU118Tweaks ++
    new WithMyVCU118System ++ 
    new CustomRocketConfig 
)

I hope my problem is clear and interesting. What are a.user.amba_prot signals and why are they undriven when I hook up both XDMA and my peripheral? Why is it that I can declare either myPeripheral or the XDMA, but, when I hook up both, these signals don't have drivers? I realize this is a difficult question with a lot of moving parts in an already scarcely viewed tag. If you took the time to read this and have suggestions, your kindness and expertise are greatly appreciated.

Edit: I think the issue might be parameter negotiation failing between Test Harness's diplomacy region and ChipTop's diplomacy region. This is the control node in XDMA.

val control = AXI4SlaveNode(Seq(AXI4SlavePortParameters(
    slaves = Seq(AXI4SlaveParameters(
      address       = List(AddressSet(c.control, c.ecamMask)),
      resources     = device.reg("control"),
      supportsWrite = TransferSizes(1, 4),
      supportsRead  = TransferSizes(1, 4),
      interleavedId = Some(0))), // AXI4-Lite never interleaves responses
    beatBytes = 4)))

This is the port the system sees.

  val mmioTLNode = TLManagerNode(
    mmioPortParamsOpt.map(params =>
      TLSlavePortParameters.v1(
        managers = Seq(TLSlaveParameters.v1(
          address            = AddressSet.misaligned(params.base, params.size),
          resources          = device.ranges,
          executable         = params.executable,
          supportsGet        = TransferSizes(1, sbus.blockBytes),
          supportsPutFull    = TransferSizes(1, sbus.blockBytes),
          supportsPutPartial = TransferSizes(1, sbus.blockBytes))),
        beatBytes = params.beatBytes)).toSeq)

  mmioPortParamsOpt.map { params =>
    sbus.coupleTo(s"port_named_$portName") {
      (mmioTLNode
        := TLBuffer()
        := TLSourceShrinker(1 << params.idBits)
        := TLWidthWidget(sbus.beatBytes)
        := _ )
    }
  }

param.beatBytes is currently set to 8 (site(MemoryBusKey).beatBytes). But the control configuration slave node is 4.