Flink streaming job won't connect to localstack s3

user_10779082 0
Viewed 416

I'm using a product called Localstack to mock Amazon S3 locally, which is serving as a streaming file sink for a Flink job.

In the run logs, I can see that Flink disregards Localstack and attempts to contact Amazon S3.

Received error response: org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Service Unavailable

Retrying Request: HEAD https://s3.amazonaws.com /testBucket/

In a flink-conf.yaml, I've specified the following configuration properties:

s3.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
s3.buffer.dir: ./tmp                           
s3.endpoint: localhost:4566
s3.path.style.access: true
s3.access-key: ***
s3.secret-key: ***

Why might Flink disregard the s3.endpoint?

localstack flink-streaming apache-flink amazon-s3
1 Answers

You are almost right with your config, you will need to add http when using localstack:

s3.endpoint: http://localhost:4566

and maybe try with extra dummy secrets as environment variables:

AWS_ACCESS_KEY_ID=foo
AWS_SECRET_ACCESS_KEY=bar
user_61995240
Related