How can I set up a schedule to continuously build a dataset until it aborts?

domdomegg 0
Viewed 46

I have a dataset which I want to continuously build until it aborts. The pseudocode of my transform looks something like:

def my_transform(input, output):
  antijoin output onto input to remove rows we've already processed
  if there are no rows left in the input, abort
  otherwise process the first 100 rows and write them to the output as an append

We can't do all the processing in a single build because there are sometimes too many rows to process and the build would fail if we tried to process all of them.

Is there some way to set up a schedule so that if there is new output, it builds the output dataset again (as if the dataset was both a trigger and target dataset)?

palantir-foundry
1 Answers

Schedule with each Minute? it only builds if there is new data. if the dataset is to large, try to split it via code repository? or of course to include a column time_created? and use this as marker?

As usual i miss functionality of a database in Foundry.

Marco0
Related