How to use external proxy for a web scraper deployed on gke cluster?

user_15416527 0
Viewed 232

I have a web scraping app that is deployed on a kubernetes cluster on GCP and this app uses a proxy to connect to multiple browsers. So I need to assign a static ip address so that the proxy can do its job, but the problem here is that I can't figure out which address to give to the proxy server. I tried to give the ingress and egress static address but it always shows "Connection refused with the proxy" error.

PS: The proxy works like magic on my local machine as I'm using a dns server to fix the dynamic IP addresses situation.

How should I do the same when the environment is actually on GKE.

google-kubernetes-engine web-scraping google-cloud-platform dns proxy
1 Answers

As your traffic comes into the cluster from single point you should be diverting the traffic from single point also which is Egress point.

I am not sure what you mean when you say egress but it will be Node IP mostly if you are running GKE default.

Kubernetes uses the Node IP as outgoing IP on which POD is scheduled.

I would suggest setting up NAT gateway in front of the GKE cluster. So all your traffic goes from a single point and you can Whitelist this IP into your Proxy and it will work.

Now if you are not much into GKE and NAT set up just this Terraform which will do all the work for you. Terraform will set up the NAT gateway.

Terraform NAT GKE : https://registry.terraform.io/modules/GoogleCloudPlatform/nat-gateway/google/latest/examples/gke-nat-gateway

Github Repo : https://github.com/GoogleCloudPlatform/terraform-google-nat-gateway/tree/v1.2.3

enter image description here

user_55258240
Related