How do I handle a no space left error in AWS Glue DataBrew?

user_10335909 0
Viewed 383

When running for 1 month's worth of data I get the below errors. In the advanced settings I have the maximum number of units set to the maximum of 10. What is the recommended approach to handle this?

An error occurred while calling o62075.collectToPython. No space left on device

An error occurred while calling o62081.pyWriteDynamicFrame. /tmp/blockmgr-9adbd7f0-0068-4244-bf85-3c59df19b610/33/temp_shuffle_4ec99ff1-e730-46ab-8b94-e35d51308aa6 (No space left on device)
aws-glue amazon-web-services
1 Answers

I think you are in the right direction about the units. Depending on your dataset size, the recommended approach is to increase the number of units for a given job. For all subsequent job runs, DataBrew allocate that number of units for processing.

Maximum you can allocate is 149 units. Link to official docs - here

user_48445570
Related