Workers started successfully dataflow flex template but sanbox stopped reduced the number of workers with autoscaling

Question

I have a dataflow flex template and started a run (gcloud flex-template run...) with 3 workers. This workers started successfully but the sandbox launcher stops and my autoscalling resized the workers to 1, as the image below:

Logging Google Cloud Dataflow flex-template

After this, the job runs normally with 1 worker and autoscalling resize the number of workers normally. Why it's happenning? I really need that my job start with 3 workers and autoscale if needs

This is my dockerfile:

FROM gcr.io/dataflow-templates-base/python3-template-launcher-base

RUN apt-get update -y && \
    apt-get install -y wget && \
    apt-get install -y git




#Copy all files in current directory and send to /dataflow/template
COPY . /dataflow/template

#Define the /app directory as work directory
WORKDIR /dataflow/template

ENV DATAFLOW_PYTHON_COMMAND_SPEC="/dataflow/template/python_command_spec.json"
ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="/dataflow/template/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_SETUP_FILE="/dataflow/template/setup.py"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="/dataflow/template/main.py"


#INSTALL REQUIREMENTS
#install apache-beam[gcp] outside of the requirements.txt to speed up the launch container
RUN apt-get update \
    && apt-get install -y libffi-dev git \
    && rm -rf /var/lib/apt/lists/* \
    # install the requirements and upgrade pip
    && pip install --no-cache-dir --upgrade pip \
    && pip install keyring \
    && pip install keyrings.google-artifactregistry-auth \
    && pip install --no-cache-dir apache-beam[gcp]==2.38.0 \
    && pip install --no-cache-dir -r $FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE \
    # Download the requirements to speed up launching the Dataflow job.
    && pip download --no-cache-dir --dest /tmp/dataflow-requirements-cache -r $FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE


#Since we already downloaded all the dependencies, there's no need to rebuild everything.
ENV PIP_NO_DEPS=True

user_582333 · Answer

Once your job is launched, there is no need to keep the sandbox up. The fact that it's adjacent to the autoscaling log is just a coincidence--it determined that there was barely any work left to do for this stage and so it downsized the pool.

Are you seeing different behavior if you launch this pipeline from a template vs. manually? If not, it may be easier to debug the issue without templates in the loop regardless of your final method of deployment.

Workers started successfully dataflow flex template but sanbox stopped reduced the number of workers with autoscaling

1 Answers