Hadoop s3 configuration file missing

user_13635464 0
Viewed 4109

I'm using the Hadoop library to upload files in S3. Because of some metric configuration file is missing I'm getting this exception

MetricsConfig - Could not locate file hadoop-metrics2-s3a-file-system.properties org.apache.commons.configuration2.ex.ConfigurationException: 

Could not locate: org.apache.commons.configuration2.io.FileLocator@77f46cee[fileName=hadoop-metrics2-s3a-file-system.properties,basePath=<null>,sourceURL=,encoding=<null>,fileSystem=<null>,locationStrategy=<null>]

My current configurations are

configuration.set("fs.s3a.access.key", "accessKey")
configuration.set("fs.s3a.secret.key", "secretKey")

Where to add this configuration file? What to add to that configuration file?

amazon-web-services hadoop amazon-s3 scala
1 Answers

don't worry about it, it's just an irritating warning. It's only relevant when you have the s3a or abfs connectors running in a long-lived app where the metrics are being collected and fed to some management tooling.

Set the log level to warn in the log4j.properties file in your spark conf dir

log4j.logger.org.apache.hadoop.metrics2=WARN
user_22612740
Related