Running nuget restore twice, second restore will not load nuget.config

I need to run a nuget restore twice in the same job in an Azure DevOps pipeline. I'm using a pipeline decorator to inject some steps at the end of the job. The pipeline will call nuget restore to build and the decorator will call restore to retrieve scripts it needs.

These are the nuget tasks in the primary pipeline:

- task: NuGetToolInstaller@1
  displayName: 'Use NuGet'
  inputs:
    versionSpec: 5.8
    checkLatest: true

- task: DotNetCoreCLI@2
  displayName: 'dotnet restore'
  inputs:
    command: 'restore'
    projects: '**/*.csproj'
    feedsToUse: config
    nugetConfigPath: .nuget/nuget.config

And here's the tasks in the pipeline decorator:

- task: NuGetToolInstaller@1
  displayName: (DECORATOR) Nuget Install
  inputs:
    versionSpec: 6.3.x

- task: NuGetCommand@2
  displayName: (DECORATOR) Nuget Restore
  inputs:
    command: 'restore'
    restoreSolution: './Decorator/packages.config'
    feedsToUse: 'config'
    nugetConfigPath: ./Decorator/nuget.config
    packagesDirectory: './Scripts'

If I comment out the first restore, the second one will succeed. That rules out a pathing issue. Both nuget.configs are being written to separate folders.

I also tried running the following commands prior to running my nuget restore, but did not resolve:

  nuget locals all -list
  nuget locals all -clear

If I do not comment the first restore the second restore fails. The error indicates it can't find my package. It's only looking at api.nuget.org so it must not be loading the nuget.config stored in the decorator folder which contains the packageSource pointing to my feed.

Also, it's not the multiple NugetToolInstaller tasks installing different versions. I only commented out the restore task in my testing and it still worked.

It should be noted the primary pipeline code is just for testing purposes. In the real world I won't have any control over what others put in their pipelines. I have control over the decorator so I need a way to load my scripts that won't interfere or cause the pipeline to fail. I'd also entertain another method besides nuget if this is not a good approach. I'm at step 1, trying to understand what's causing the failure with nuget.

Any ideas?