How to hide/mask S3 credentials from druid logs

Viewed 102

We have populated S3 credentials in the job.properties of druid ingestion script as mentioned below.

"jobProperties" : {
  "fs.s3a.impl" : "org.apache.hadoop.fs.s3a.S3AFileSystem",
  "fs.AbstractFileSystem.s3a.impl" : "org.apache.hadoop.fs.s3a.S3A",
  "fs.s3a.access.key" : "YOUR_ACCESS_KEY",
  "fs.s3a.secret.key" : "YOUR_SECRET_KEY"
}

However, this secret and access key is printed in the logs and is causing privacy/security issues. Could you please let me know how to hide/mask the credentials.

I have tried the below as well...but didnt work.

export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY
export AWS_DEFAULT_REGION=
  "jobProperties" : {
  "fs.s3a.impl" : "org.apache.hadoop.fs.s3a.S3AFileSystem",
  "fs.AbstractFileSystem.s3a.impl" : "org.apache.hadoop.fs.s3a.S3A",
  "fs.s3.awsAccessKeyId": {
               "type": "environment",
               "variable": "AWS_ACCESS_KEY_ID"
              },
  "fs.s3.awsSecretAccessKey": {
              "type": "environment",
              "variable": "AWS_SECRET_ACCESS_KEY"
             },
   "fs.s3.impl": "org.apache.hadoop.fs.s3native.NativeS3FileSystem",
        "fs.s3n.awsAccessKeyId":{
               "type": "environment",
               "variable": "AWS_ACCESS_KEY_ID"
              },
   "fs.s3n.awsSecretAccessKey": {
              "type": "environment",
              "variable": "AWS_SECRET_ACCESS_KEY"
             }
}

Thanks

0 Answers
Related