EMR dyanmodb export failed because of table capacity set to on-demand

user_5921634 0
Viewed 1836

After we changed the dynamodb table capacity to on-demand, the data pipeline job to export dynamodb table failed with this error. 

Exception in thread "main" java.lang.RuntimeException: Read throughput should not be less than 1. Read throughput percent: 0.0
at org.apache.hadoop.dynamodb.read.AbstractDynamoDBInputFormat.getSplits(AbstractDynamoDBInputFormat.java:51)
at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)

Any workaround to this issue?

Thanks

--gsu

amazon-dynamodb amazon-web-services
2 Answers

I'd contact AWS support to confirm, but I was told the EMR DynamoDB connector does not formally support tables using on-demand provisioning yet. So, more than likely you need to switch the table back to provisioned capacity as a workaround.

Edit: As of 23 January 2019, the EMR connector for DynamoDB supports tables set to on-demand billing.

user_34161350

If the issue is not resolved, then you might need to change at 3 places:

  1. You need to use emr release emr-5.26.0 to emr-5.30.0.
  2. Replace org.apache.hadoop.dynamodb.tools.DynamoDbExport with org.apache.hadoop.dynamodb.tools.DynamoDBExport. Notice the change in casing. Similar for DynamoDBImport.
  3. If you are using emr-dynamodb-connector, then you need to clone its latest version, generate the emr-ddb-tools jar by mvn clean install and then use the generated jar for emr-dynamodb-tools which would be version 4 or higher currently and replace this in your arguments. These should resolve the issue.

Also, there is an issue currently with emr releases 31 or higher if you are using emr-dynamodb-tools which shows you error related to some joda-time framework. I would restrict to using any releases between emr-5.26.0 to emr-5.30.0.

user_75185610
Related