I need to configure an ec2 instance as a scrape to monitor all instances created in a region. I understand that the scrape can monitor instances by the tag or by the instance id, so I have created the following configuration.
global:
scrape_interval: 15s
external_labels:
monitor: 'prometheus'
scrape_configs:
- job_name: test-dev
ec2_sd_configs:
- region: us-west-2
secret_key: "xxxxxxxxxx"
access_key: "xxxxxx"
port: 9404
filters:
- name: instance-id
values:
- i-xxxxxxxxxxxxxx
remote_write:
- url: https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-xxxxxxxxxxx-b0d3-4cfe-b04d-xxxxxxxxxx/api/v1/remote_write
queue_config:
max_samples_per_send: 1000
max_shards: 200
capacity: 2500
I am using as a remote write workspace created on Amazon Prometheus through an IAM role with AmazonEC2ReadOnlyAccess and AmazonPrometheusRemoteWriteAccess permissions.
when running prometheus I see the following error:
ts=2022-08-09T20:07:45.931Z caller=dedupe.go:112 component=remote level=error remote_name=662c9f url=https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-xxxxxxxx-b0d3-4cfe-b04d-xxxxxxxxxx/api/v1/remote_write msg="non-recoverable error" count=5 exemplarCount=0 err="server returned HTTP status 403 Forbidden: {\"message\":\"Missing Authentication Token\"}"
any helps??