Communication issues with Python gRPC client and server in k8s cluster

Question

Update: On my client application, I used the server's IP address directly and it worked. Now, I need to figure out why I am unable to connect using DNS names.

Update #2: After following DazWilkin's suggestions, I was able to establish a connection using an insecure channel between client and server without relying on IP addresses. This is what I did:

I removed the k8s ingress from the equation and instead used the server's service DNS name: {service}.{namespace}.svc.cluster.local
I set my server application's k8s service's port and targetPort to use the port running the gRPC server (9000)
On my client, I set the GRPC_SERVER_ADDRESS variable as: :

Original Post

I am trying to create a simple gRPC client/server in Python, but I am unable to establish a connection between the two parts. Both my client and server applications are running in separate k8s namespaces within the same network. Both also have an ingress, a service, and a dnsservice for communication purposes.

Obs.: The k8s service on both applications maps port 80 to port 9000.

Here's the relevant snippet of code for my client:

# client.py

GRPC_SERVER_ADDRESS = "grpc-server.com"


@image_bp.route('/image-grpc', methods=['POST'])
def modify_image_grpc():
    request_json = request.get_json()
    with grpc.insecure_channel(GRPC_SERVER_ADDRESS) as channel:
        stub = image_pb2_grpc.ImageServiceStub(channel)
        image = stub.GetImage(image_pb2.ImageRequest(
            id_user=request_json['id_user'],
        ))
    
    # response = ...

    return response

And here is the relevant code snippet for my server:

# server.py

GRPC_MAX_WORKERS = 10
GRPC_HOST = "0.0.0.0"
GRPC_PORT = "9000"


def serve():
    try:
        server = grpc.server(futures.ThreadPoolExecutor(max_workers=GRPC_MAX_WORKERS))
        image_pb2_grpc.add_ImageServiceServicer_to_server(ImageService(), server)
        server.add_insecure_port(f'{GRPC_HOST}:{GRPC_PORT}')
        server.start()
        server.wait_for_termination()
    except Exception as e:
        logger.error(f'[-] Error serving gRPC on port {GRPC_PORT}: {e}')
        sys.exit()

And here is the error message produced in my client application whenever I try to send a request:

# client.error.log

<_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1662987231.739227750","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file
_line":3260,"referenced_errors":[{"created":"@1662987231.739227058","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_lin
e":167,"grpc_status":14}]}"
>

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)

... (line in my code that raises the exception: stub.GetImage)

  File "/usr/local/lib64/python3.6/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib64/python3.6/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1662987231.739227750","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file
_line":3260,"referenced_errors":[{"created":"@1662987231.739227058","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_lin
e":167,"grpc_status":14}]}"

Package versions:

# requirements.txt

...

grpcio==1.47.0
grpcio-tools==1.47.0

...

I have seen some solutions online that utilize a secure channel, but I intend on using an insecure channel. Any ideas?

Communication issues with Python gRPC client and server in k8s cluster

0 Answers