I have Kubernetes set up and running a grpc service in a pod. I am successfully hitting an endpoint on the service, which has a print() statement in it, but I see no logs in the log file. I have seen this before when I was running a (cron) job in Kubernetes and the logs only appeared after the job was done (as opposed to when the job was running). Is there a way to make kubernetes write to the log file right away? Any setting that I can put (either cluster-level or just for the pod)? Thanks for any help in advance!
3 Answers
Here is an example of K8S deployment yaml so you can copy paste the solution from the aforementioned answer:
root@k8s:~/python_docker$ cat deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hpa
spec:
selector:
matchLabels:
app: hpa
template:
metadata:
labels:
app: hpa
spec:
containers:
- name: hpa
image: my-hpa/py
env:
- name: PYTHONUNBUFFERED
value: "0"
resources:
requests:
cpu: 100m
memory: 200Mi
One possibility is that the container is starved for CPU. We have run into this issue when running locally on minikube with resource limits that enforced in our larger cluster. Try bumping the CPU resource limits on your pod. Below is an example yaml.
If your CPU limits are around 20-40m, that might be too low to run a full flask/python app. You might try bumping it to closer to 100m. It's not going to crush your local machine.
apiVersion: apps/v1
kind: Deployment
metadata:
name: python-app
spec:
selector:
matchLabels:
app: python-app
template:
metadata:
labels:
app: python-app
spec:
containers:
- name: python-app
image: python-app
imagePullPolicy: Never
resources:
requests:
cpu: 40m
memory: 40Mi
limits:
cpu: 20m
memory: 20Mi
Related