I have prometheus task running on ECS Fargate and an EC2 instance where I installed Influxdb to store prometheus metrics for data persistence in case of prometheus ECS task restarts.
I configured remote read and remote write URL's in prometheus.yml and deployed it.
Here when I opened 8086 port from anywhere the connectivity from ECS task to EC2 VM happening successfully and data is written into InfluxDB. But when I changed the security group to open 8086 port only for the security group where EC2 VM and ECS task are running then I am getting error as -
ts=2022-09-23T04:40:18.441Z caller=dedupe.go:112 component=remote level=warn remote_name=33b2e7 url="http://x.x.x.x:8086/api/v1/prom/write?db=prometheus" msg="Failed to send batch, retrying" err="Post \"http:/x.x.x.x:8086/api/v1/prom/write?db=prometheus\": context deadline exceeded"
Both ECS cluster and EC2 VM are in same VPC and same security group.