Error with multiple Vcenter connection in my powershell script

I have a PowerShell script that needs to start a connection on two Vcenters. So the beginning of my script is:

Set-PowerCLIConfiguration -DefaultVIServerMode Multiple -Confirm:$false

Connect-VIServer -Server Vcenter1 -User XXXX -Password XXX

Connect-VIServer -Server Vcenter2 -User XXXX  -Password XXXX 

Now, in my script I have a condition:

if(element find in Vcenter1){
Get-VM my_VM| New-TagAssignment -Tag XXXX -Server Vcenter1

}

elseif(element find in Vcenter2){
 Get-VM my_VM| New-TagAssignment -Tag XXXX -Server Vcenter2
}

But my script return this error:

New-TagAssignment 
The TagCisImp - 'my_tag' must be managed by the same VC Server that you are using to invoke this operation

I don't understand why I am getting this error, even if I set the ServerMode to multiple and with each -Server parameters...

Can someone tell me what is wrong with my script?