How can i get the branch name that is merged into target branch that trigger the pipeline?
You could use the predefined variables System.PullRequest.SourceBranch, which return the branch that is being reviewed in a pull request.

As test, I add a inline powershell task to output this variable:
Write-Host "The merged target branch is:-" $(System.PullRequest.SourceBranch)
Then we could get the value is refs/heads/1805.
We could use a command line task to parse that path to get the branch name:
set BranchName=$(System.PullRequest.SourceBranch)
for %%a in (%BranchName%) do echo %%~nxa

Update:
As I test, it works fine on my side with following code:
Write-Host "##vso[task.setvariable variable=SourceBranchName;]$(System.PullRequest.SourceBranch)"
Remove the isOutput=true.
Then, I could get the variable SourceBranchName by the command line task:
echo $(SourceBranchName)

According to your error, it seems the error comes from your powershell task itself or the agent. So, please try to use hosted agent with Inline powershell task to check if you still have this issue.
Hope this helps.