Kubernetes "logs" does not show any output for a simple python program running in a container in a pod

user_10991905 0
Viewed 2443

I have a very simple setup. My .yaml-Configuration file for Kubernetes:

apiVersion: batch/v1
kind: Job
metadata:
  name: stdout-test
spec:
  template:
    spec:
      priorityClassName: research-high
      containers:
        - name: container-stdout-test
          image: <here comes my secret image repo>
          imagePullPolicy: "IfNotPresent"
          resources:
            limits:
              nvidia.com/gpu: "1"
              cpu: "1"
              memory: "8Gi"
            requests:
              nvidia.com/gpu: "1"
              cpu: "1"
              memory: "4Gi"
          command: ["python3", "/workspace/main.py"]
          volumeMounts:
            - mountPath: /workspace 
              name: localdir 
      imagePullSecrets:
        - name: lsx-registry
      restartPolicy: "Never"
      volumes:
        - name: localdir
          cephfs:
            monitors:
              - <here come my secret monitors>
            user: <namespace>
            path: "/home/stud/nothelfer/stdout-test" 
            secretRef: 
              name: <my secret>

And my simple python program (main.py):

import time

for i in range(0, 1000):
    print(i)
    time.sleep(1.0)

Starting and running the job in kubernetes works fine. I get this output from kubectl get pods:

NAME                READY   STATUS    RESTARTS   AGE
stdout-test-hl6qs   1/1     Running   0          5s

But i dont get the expected output 0, 1, 2, ... from stdout by using kubectl logs -f stdout-test-hl6qs. Instead i just get a blank screen. There is just no output printed to the console. I was expecting to get the output of my simple python program. I have tried all possible command line arguments for kubectl logs but none of them make me see the output of my Python program in the command line. The image is set up correctly, the Python program runs fine in the container, I have already checked all that. Can anyone help me here?

My Dockerfile:

FROM pytorch/pytorch
RUN conda install matplotlib pandas numpy
RUN conda update --all

ENV PYTHONUNBUFFERED 1

RUN python -c "import torch, pandas, matplotlib"
kubernetes logging python
4 Answers

After quite a few tries, I figured out how to see the stdout output. In the Kubernetes configuration file, I had to replace line command: ["python3", "/workspace/main.py"] with command: ["python3", "-u", "/workspace/main.py"], which configures the output from Python to unbuffered. For reasons that aren't entirely clear, it didn't work when I only set variable ENV PYTHONUNBUFFERED=1 or ENV PYTHONUNBUFFERED 1 in the Dockerfile. I can now see the output of the program whether I specified ENV PYTHONUNBUFFERED=1 or ENV PYTHONUNBUFFERED 1 in the Dockerfile or not.

PYTHONUNBUFFERED

user_109919050

Add the following line in your Dockerfile:

ENV PYTHONUNBUFFERED=1

See this

user_104291890

What helped me see my logs in the k8s logs (using Lens) was to add the following after my execution statement |& tee /proc/1/fd/1

So you would basically have something like this

$ python some_python_program.py |& tee /proc/1/fd/1
user_99174030
user_12714120
Related