Wrong connection port despite Kubernetes deployments/services ports specified

Question

It might take a while to explain what I'm trying to do but bear with me please.

I have the following infrastructure specified:

I have a job called questo-server-deployment (I know, confusing but this was the only way to access the deployment without using ingress on minikube)

This is how the parts should talk to one another:

And here you can find the entire Kubernetes/Terraform config file for the above setup

I have 2 endpoints exposed from the node.js app (questo-server-deployment) I'm making the requests using 10.97.189.215 which is the questo-server-service external IP address (as you can see in the first picture)

So I have 2 endpoints:

health - which simply returns 200 OK from the node.js app - and this part is fine confirming the node app is working as expected.
dynamodb - which should be able to send a request to the questo-dynamodb-deployment (pod) and get a response back, but it can't.

When I print env vars I'm getting the following:

➜ kubectl -n minikube-local-ns exec questo-server-deployment--1-7ptnz -- printenv
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=questo-server-deployment--1-7ptnz
DB_DOCKER_URL=questo-dynamodb-service
DB_REGION=local
DB_SECRET_ACCESS_KEY=local
DB_TABLE_NAME=Questo
DB_ACCESS_KEY=local
QUESTO_SERVER_SERVICE_PORT_4000_TCP=tcp://10.97.189.215:4000
QUESTO_SERVER_SERVICE_PORT_4000_TCP_PORT=4000
QUESTO_DYNAMODB_SERVICE_SERVICE_PORT=8000
QUESTO_DYNAMODB_SERVICE_PORT_8000_TCP_PROTO=tcp
QUESTO_DYNAMODB_SERVICE_PORT_8000_TCP_PORT=8000
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
QUESTO_SERVER_SERVICE_SERVICE_HOST=10.97.189.215
QUESTO_SERVER_SERVICE_PORT=tcp://10.97.189.215:4000
QUESTO_SERVER_SERVICE_PORT_4000_TCP_PROTO=tcp
QUESTO_SERVER_SERVICE_PORT_4000_TCP_ADDR=10.97.189.215
KUBERNETES_PORT_443_TCP_PROTO=tcp
QUESTO_DYNAMODB_SERVICE_PORT_8000_TCP=tcp://10.107.45.125:8000
QUESTO_DYNAMODB_SERVICE_PORT_8000_TCP_ADDR=10.107.45.125
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
QUESTO_SERVER_SERVICE_SERVICE_PORT=4000
QUESTO_DYNAMODB_SERVICE_SERVICE_HOST=10.107.45.125
QUESTO_DYNAMODB_SERVICE_PORT=tcp://10.107.45.125:8000
KUBERNETES_SERVICE_PORT_HTTPS=443
NODE_VERSION=12.22.7
YARN_VERSION=1.22.15
HOME=/root

so it looks like the configuration is aware of the dynamodb address and port:

QUESTO_DYNAMODB_SERVICE_PORT_8000_TCP=tcp://10.107.45.125:8000

You'll also notice in the above env variables that I specified:

DB_DOCKER_URL=questo-dynamodb-service

Which is supposed to be the questo-dynamodb-service url:port which I'm assigning to the config here (in the configmap) which is then used here in the questo-server-deployment (job)

Also, when I log:

kubectl logs -f questo-server-deployment--1-7ptnz -n minikube-local-ns

I'm getting the following results:

Which indicates that the app (node.js) tried to connect to the db (dynamodb) but on the wrong port 443 instead of 8000?

The DB_DOCKER_URL should contain the full address (with port) to the questo-dynamodb-service

What am I doing wrong here?

Edit ----

I've explicitly assigned the port 8000 to the DB_DOCKER_URL as suggested in the answer but now I'm getting the following error:

Seems to me there is some kind of default behaviour in Kubernetes and it tries to communicate between pods using https ?

Any ideas what needs to be done here?

user_14704799 · Answer

How about specify the port in the ConfigMap:

...
data = {
  DB_DOCKER_URL = ${kubernetes_service.questo_dynamodb_service.metadata.0.name}:8000
...

Otherwise it may default to 443.

user_1976982 · Answer

Answering my own question in case anyone have an equally brilliant idea of running local dybamodb in a minikube cluster.

The issue was not only with the port, but also with the protocol, so the final answer to the question is to modify the ConfigMap as follows:

data = {
    DB_DOCKER_URL = "http://${kubernetes_service.questo_dynamodb_service.metadata.0.name}:8000"
    ...
}

As a side note:

Also, when you are running various scripts to create a dynamodb table in your amazon/dynamodb-local container, make sure you use the same region for both creating the table like so:

#!/bin/bash

aws dynamodb create-table \
    --cli-input-json file://questo_db_definition.json \
    --endpoint-url http://questo-dynamodb-service:8000 \
    --region local

And the same region when querying the data.

Even though this is just a local copy, where you can type anything you want as a value of your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and actually in the AWS_REGION as well, the region have to match.

If you query the db with a different region it was created with, you get the Cannot do operations on a non-existent table error.

Wrong connection port despite Kubernetes deployments/services ports specified

2 Answers