Grpc connection in kubernetes is not working =>UNAVAILABLE: Network closed for unknown reason

user_2054224 0
Viewed 3782

I have a problem with grpc communication on kubernetes. I have a java client calling a simple helloworld grpc service (in java too). Everything works fine in Docker but not in kubernetes :'(.

You can get the whole code here: https://github.com/hagakure/testgrpc

How to reproduce?

  • just run ./start.bat
  • then I create my deployments and I expose them on kubectl.
  • You can access the service with this address: http://localhost:8080/hello?firstName=cedric but it doesn't work :'(
  • I added a second method (http://localhost:8080/check?address=host.docker.internal&port=8081) to try different address/port combination to connect to server but yet it never works.
  • I also tried a simple http connection between two exposed services and that works fine so it's not just the ip/port that doesn't match :(

If you have any clue on how to make grpc works in kubernetes, thanks for your help.

grpc-java kubernetes docker java
1 Answers
  • Your deployment yamls are lacking the service part to route the traffic to the pods appropriately.
  • Also, the deployments are not passing the Environment Variables as specified in the docker-compose.yml. Here are the fixed yamls.
  • deployment_server.yml:
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: grpc-server
  name: grpc-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: grpc-server
  template:
    metadata:
      labels:
        app.kubernetes.io/name: grpc-server
    spec:
      containers:
      - image: poc_grpc-server:latest
        imagePullPolicy: Never
        name: grpc-server
        ports:
        - containerPort: 8081
        env:
        - name: GRPC_SERVER_PORT
          value: "8081"
---
apiVersion: v1
kind: Service
metadata:
  name: grpc-server-svc
spec:
  selector:
    app.kubernetes.io/name: grpc-server
  ports:
    - protocol: TCP
      port: 8081
  • deployment_client.yml:
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: grpc-client
  name: grpc-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: grpc-client
  template:
    metadata:
      labels:
        app.kubernetes.io/name: grpc-client
    spec:
      containers:
      - image: poc_grpc-client
        imagePullPolicy: Never
        name: grpc-client
        ports:
        - containerPort: 8080
        env: 
          - name: GRPC_SERVER_ADDRESS
            value: "grpc-server-svc"
          - name: GRPC_SERVER_PORT
            value: "8081"
---
apiVersion: v1
kind: Service
metadata:
  name: grpc-client-svc
spec:
  selector:
    app.kubernetes.io/name: grpc-client
  ports:
    - protocol: TCP
      port: 8080
  type: NodePort

Highlights:

  • created grpc-server-svc service as ClusterIP to be available only inside the cluster serving the GRPC Server.
  • created grpc-client-svc service to NodePort to demonstrate receiving curl requests from outside the cluster.
  • Added the env section: Note that I set GRPC_SERVER_ADDRESS to grpc-server-svc since we are not in docker environment anymore host.docker.internal is no longer an option we are now targetting the service mentioned above.
  • I'm using imagePullPolicy set to Never only for this example, since I'm using my local docker registry.

Reproduction:

  • After building the images:
$ docker image ls
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
poc_grpc-client                           latest              7f6d886a1612        24 minutes ago      660MB
poc_grpc-server                           latest              d46bf9481d1c        24 minutes ago      658MB
  • Deployed the fixed yamls as above:
$ kubectl apply -f deployment_server.yml 
deployment.apps/grpc-server created
service/grpc-server-svc created

$ kubectl apply -f deployment_client.yml 
deployment.apps/grpc-client created
service/grpc-client-svc created

$ kubectl get all
NAME                               READY   STATUS    RESTARTS   AGE
pod/grpc-client-6ffcf6b6c8-846s5   1/1     Running   0          3s
pod/grpc-server-5d7fd9cb89-dkqlb   1/1     Running   0          7s

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/grpc-client-svc   NodePort    10.99.58.76    <none>        8080:32224/TCP   3s
service/grpc-server-svc   ClusterIP   10.96.67.139   <none>        8081/TCP         7s
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP          3h36m

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grpc-client   1/1     1            1           3s
deployment.apps/grpc-server   1/1     1            1           7s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/grpc-client-6ffcf6b6c8   1         1         1       3s
replicaset.apps/grpc-server-5d7fd9cb89   1         1         1       7s
  • Now I'll curl the client, since I'm running on minikube (1 node cluster) the IP of the node is the IP of the cluster, I'll pair the IP with the NodePort assigned.
$ kubectl cluster-info
Kubernetes master is running at https://172.17.0.4

$ curl http://172.17.0.4:32224
Client is running!!!

$ curl http://172.17.0.4:32224/hello?firstName=Cedric
Hello Cedric

If you have any questions, let me know in the comments!

user_125241590
Related