Error when kicking off a Cloud Run service

Question

i am working on a GCP dataflow project which creates flex templates and it is being run using cloud build. I am attempting, instead of using cloud build, to run via cloud run but i am getting some dodgy error. I am suspecting the way i have structured my Dockerfile makes incompatible for it to be used for cloudbuild and cloudrun at the same time. Below is my dockerfile

FROM gcr.io/dataflow-templates-base/python3-template-launcher-base

ARG WORKDIR=/dataflow/template
RUN mkdir -p ${WORKDIR}
RUN mkdir -p ${WORKDIR}/modules
WORKDIR ${WORKDIR}

COPY spec/python_command_spec.json ${WORKDIR}/python_command_spec.json
COPY modules ${WORKDIR}/modules

ENV DATAFLOW_PYTHON_COMMAND_SPEC ${WORKDIR}/python_command_spec.json

RUN pip install avro-python3 pyarrow==0.15.1 apache-beam[gcp]==2.27.0  pandas-datareader==0.9.0
RUN apt-get update && apt-get install -y iputils-ping dnsutils 

COPY __init__.py ${WORKDIR}/__init__.py
COPY setup.py ${WORKDIR}/setup.py
COPY main.py ${WORKDIR}/main.py



# Super important to add these lines.
ENV FLEX_TEMPLATE_PYTHON_SETUP_FILE="${WORKDIR}/setup.py"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/main.py"
#RUN python ${WORKDIR}/setup.py install

RUN echo '----- listing workdir'
RUN ls -la ${WORKDIR}
RUN echo '-- pinging google--'
RUN echo '-- pinging sec.gov'

RUN echo '--- bulding image---'


ENTRYPOINT [ "/opt/apache/beam/boot" ]

the last line allows me to run my template using the following yaml

substitutions:
  _IMAGE: my_logic:latest2
  _JOB_NAME: 'pipelinerunner'
  _TEMP_LOCATION: ''
  _REGION: us-central1
  _FMPKEY: ''  
  
steps:
- name: gcr.io/$PROJECT_ID/$_IMAGE
  entrypoint: python
  args:
  - /dataflow/template/main.py
  - --runner=DataflowRunner
  - --project=$PROJECT_ID
  - --region=$_REGION
  - --job_name=$_JOB_NAME
  - --temp_location=$_TEMP_LOCATION
  - --sdk_container_image=gcr.io/$PROJECT_ID/$_IMAGE
  - --disk_size_gb=50
  - --year=2018 
  - --quarter=QTR1 
  - --fmpkey=$_FMPKEY
  - --setup_file=/dataflow/template/setup.py  

options:
  logging: CLOUD_LOGGING_ONLY

However, using the docker above, whenever i kick off a cloudrun i am getting this exception

terminated: Application failed to start: Failed to create init process: failed to load /opt/apache/beam/boot: no such file or directory

What am i missing? regards Marco

Error when kicking off a Cloud Run service

0 Answers