Use different assets for APKs and app bundles

I am using Play Asset Delivery with install-time assets which is working just fine with app bundles. However, I still need to build APKs where I need regular assets. In my app's build.gradle, I tried to configure it in this way:

android {
    sourceSets {
        main {
            assets.srcDirs = ['../assets/src/main/assets']
        }
    }
}

Unfortunately, the app bundle then contains this second pair of assets besides the Play Asset Delivery asset, which leads to conflicts. How can I set assets.srcDirs only for APKs but not for app bundles?