Connecting to 2 ACRs from a custom pipeline task with a single docker-compose file

I am updating a custom Azure pipeline task using Typescript. The original task pulled images from a single ACR and now I want to pull images from another one.

I am doing it by first creating a token for authentication infront of the ACR:

registryAuthenticationToken = new ACRAuthenticationTokenProvider(tl.getInput("azureSubscriptionEndpoint"), tl.getInput("azureContainerRegistry")).getAuthenticationToken();

After that I am opening a connection to the ACR by calling the open method of this class (hostEndpoint is empty here):

export default class DockerComposeConnection extends ContainerConnection 
{
    ...
    public open(hostEndpoint?: string, authenticationToken?: AuthenticationToken): any 
    {
       super.open(hostEndpoint, authenticationToken);
       ... 
    }

and in the end I am invoking the docker-compose up command.

This is working for a single ACR. Now with a 2nd ACR and a single docker-compose file which contains all services together from both ACRs, I'm not sure how to address the issue of downloading from both of them.

I have tried creating an additional token just like I did above, but for the 2nd ACR, and then I tried opening another connection in parallel to the first one and continued everything else the same. When doing that I am receiving an error saying:

unauthorized: authentication required

I get this error even if the docker-compose file I'm working on doesn't contain any images from the 2nd ACR, meaning the connection itself is already failing and it fails the entire task.

Is my flow for trying to pull images from 2 ACRs with a single docker compose file correct? Maybe I need to do something differently like working with 2 docker compose files, one for each ACR?