Angular custom webpack with multiple targets

In my application I'm using webpack with multiple targets.

Basically I have one for the application code and one for the web worker code, i.e. sth like:

const browserConfig =  {
    output: {
        filename: "[name].bundle.js"
    },
//...
}
const workerConfig = {
    target: "webworker",
    entry: {
        decoder: "./worker-stuff.ts"
    },
    output: {
        filename: "[name].bundle.js"
    },
// ...
});

module.exports = [browserConfig, workerConfig];

Now I'm trying to integrate my application into an Angular application as a component using custom webpack configuration The browserConfig part is already handled by Angulars internal WebPack configuration so I just need to append the workerConfig. So this is how my custom webpack config looks like:

const workerConfig = {
    target: "webworker",
    entry: {
        worker: "./worker-stuff.ts"
    },
    output: {
        filename: "[name].bundle.js"
    },
// ...
});

module.exports = [workerConfig];

And I use it like this in angular.json:

          "builder": "@angular-builders/custom-webpack:browser",
          "options": {
            "customWebpackConfig": {
              "path": "./custom-webpack.config.js"
            },

Unfortunately it seems Angular can not handle arrays of configurations:

An unhandled exception occurred: Invalid configuration object. Webpack has been initialised using a configuration object that does not match the API schema.
 - configuration has an unknown property '0'. These properties are valid:
   object { amd?, bail?, cache?, context?, dependencies?, devServer?, devtool?, entry?, externals?, infrastructureLogging?, loader?, mode?, module?, name?, node?, optimization?, output?, parallelism?, performance?, plugins?, profile?, recordsInputPath?, recordsOutputPath?, recordsPath?, resolve?, resolveLoader?, serve?, stats?, target?, watch?, watchOptions? }

Removing the square brackets does also not work b/c then the Angular and my workerConfig get merged and that's not what I want.

Am I doing sth. wrong or is it just not possible to have multiple targets with custom webpack configuration?