yocto custom image apply defconfig changes to kernel build

I'm trying to apply some changes to the kernel configuration to my kernel image and to the device tree. In my custom layer my tree looks like this

meta-custom/
├── COPYING.MIT
├── README
├── classes
│   └── postprocess_image.bbclass
├── conf
│   └── layer.conf
├── recipes-core
│   └── images
│       └── custom-image.bb
├── recipes-example
│   └── example
│       └── example_0.1.bb
└── recipes-kernel
    └── linux
        ├── linux-qoriq
        │   └── ls1046ardb
        │       ├── 0001-usb0-devicetree-dtsi.patch
        │       └── fragment.cfg
        └── linux-qoriq_5.15.bbappend

and thats the content of my linux-qoriq_.bbappend file:

SRC_URI += "file://fragment.cfg"
SRC_URI += "file://0001-usb0-devicetree-dtsi.patch"
FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
PACKAGE_ARCH = "${MACHINE_ARCH}"

When I'm running bitbake custom-image during the build process .patch file and the .cfg file are copied to the folder where the linux kernel sources reside (build_ls1046ardb/tmp/work/ls1046ardb-fsl-linux/linux-qoriq) the devicetree patch get's applied correctly to the .dtsi file (build_ls1046ardb/tmp/work/ls1046ardb-fsl-linux/linux-qoriq/5.15.32+gitAUTOINC+fa6c316859-r0/git/arch/arm64/boot/dts/freescale/fsl-ls1046a.dtsi) and everything works as expected. But .cfg file changes are not applied to the build_ls1046ardb/tmp/work/ls1046ardb-fsl-linux/linux-qoriq/defconfig file and also not to the build_ls1046ardb/tmp/work/ls1046ardb-fsl-linux/linux-qoriq/build/.conf file which is used during to kernel build. I also tried to add the modified defconfig file directly to the linux-qoriq_5.15.bbappend file like this:

SRC_URI += "file://defconfig"
SRC_URI += "file://0001-usb0-devicetree-dtsi.patch"
FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
PACKAGE_ARCH = "${MACHINE_ARCH}"

But using this approach my defconfig file is not getting copied to build_ls1046ardb/tmp/work/ls1046ardb-fsl-linux/linux-qoriq/defconfig respectively overwrites the defconfig file that resides in the build_ls1046ardb/tmp/work/ls1046ardb-fsl-linux/linux-qoriq/ folder. What am I doing wrong here?