Foundry Data Storage Optimization

user_10181288 0
Viewed 83

Hi I have a general question about pipelines optimization in order to lower storage space.

Does deleting trashed datasets help alleviate disk storage? Ex. Remove obsolete datasets: a.) based on business knowledge and utilization and b.) datasets in the trash.

Also, We'd like to manage the copies of datasets that are stored when a schedule runs. We believe that if we ever had to fall back to a previous version, we only need to reference the latest one, as opposed to keeping multiple copies.

Does this affect storage? And is there a way to manage configuration on this?

palantir-foundry optimization
1 Answers

Deleting trashed datasets (in typical setups) will result in their underlying files being deleted, but typically a larger driver of storage consumption is the set of previous dataset views kept by default.

You can control the length of time these files and views are kept using the Foundry Retention service. I'd recommend you consult with platform documentation and your support team for configuration of this service.

Retention will compute and mark files matching your configuration for deletion and periodically delete them, thus reducing your storage consumption.

user12474509_149459390
Related