AWS AMP : 502 Bad Gateway: unable to proxy request - WebIdentityErr: failed to retrieve credentials

i have eks setup and provisioned aws managed service for prometheus. Created policy with AMP full access( "aps:*") and attached that policy to role which is used by EKS.

Prometheus is installed on eks but it was not able to push metrics into Prometheus managed service.

EKS is provisioned in VPC.

Error:

ts=2021-07-07T00:43:57.951Z caller=dedupe.go:112 component=remote level=warn remote_name=e595f3 url=http://localhost:8005/workspaces/ws-xxxx-xxxx-xxxx/api/v1/remote_write msg="Failed to send batch, retrying" err="server returned HTTP status 502 Bad Gateway: unable to proxy request - WebIdentityErr: failed to retrieve credentials"

ingest policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "aps:*"
            ],
            "Resource": "*"
        }
    ]
}

Trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::xxxxxx:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/65ETDGGHD56WTRSDGF"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/65ETDGGHD56WTRSDGF:sub": "system:serviceaccount:test-eks-prometheus:amp-iamproxy-query-service-account"
        }
      }
    }
  ]
}

any help on this?