Beam SDK harness still trying to launch docker when I set environment_type to be `PROCESS`

Viewed 103

According to the beam harness documentation:

PROCESS: User code is executed by processes that are automatically started by the runner on each worker node.

args = [
            "--runner=portableRunner",
            "--streaming",
            "--sdk_worker_parallelism=2",
            "--environment_type=PROCESS",
            "--environment_config={\"command\": \"/opt/apache/beam/boot\"}",
        ]
consumer_config = {
            "security.protocol": "SASL_SSL",
            "sasl.mechanism": "AWS_MSK_IAM",
            "sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;",
            "sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler",
            "bootstrap.servers": bootstrap_servers,
        }
with beam.Pipeline(options=PipelineOptions(args)) as p:
        data = p | "Reading messages from Kafka" >> ReadFromKafka(
           consumer_config=consumer_config,
           topics=topics,
           with_metadata=True
         )
         data | 'Writing to stdout' >> beam.Map(logging.info)

But when I run the code (deployed to k8s using flinkk8soperator), it is complaining:

Caused by: java.io.IOException: Cannot run program "docker": error=2, No such file or directory

Wondering if I misunderstand anything? Thanks!

1 Answers

You portable execution relies on xLang support that requires starting a Java SDK with docker. Your cluster doesn't have docker installed.

Related