I want to create the list from the datasest and use in another function in palantir foundry but not able to find any solution

user_17952470 0
Viewed 90

below code is working fine and its generating the 3 output when I am building the data sets in Palantir foundry but my problem is that u can see the 1st and 2nd line of code there I have hardcoded the data so that list I want to take it from datasets and create the list and use that list in below function so it will generate 3 output. but I tried many thing but still not able find the solution .if someone can help in this '''

gen_o =[A32,A33,A35]
program_id=[N,L,P]
def generat_each_dataSet(v_output,v_program_id):
   @transform_df(
   output("............path " + v_output)
   df =input()
   df1=input()
   )
   def compute(df,df1):
    ... here transformation code is written......
    .... in transformation i am filtering the df based on program id 
        df = df.join(df1).select()  
    
        return df
   return compute
def loop_for_each_datasets():
   transform=[]
   incnt=0
   for v_output in  gen_o
        f= generat_each_dataset(gen_o[incnt],program[incnt])
        incnt =incnt + 1
        transform.append(f)
   return transform
loop_for_each_datasets()

'''

pyspark palantir-foundry transformation sql python
1 Answers

This is a somewhat common question and it confuses the behaviour of CI time and Build time within Foundry

Your datasets and code outside the transform, is executed during the CI checks, I think it's during the renderShrinkWrap step. Inside the CI you don't have access to any Datasets so you can't use the contents of one dataset to generate transforms.

To guarantee reproducible builds, the transforms generated and related jobSpecs must be deterministic, because of this, any code that generates transforms must rely solely on things accessible through your code outside of transforms. Such has local constants which is what you have done.

You can think about it like this:

1- You commit to master

2- The CI task discovers which transform datasets your code is generating

3- The CI creates any non existing Output datasets discovered in 2

4- The CI attaches jobspecs with logic to run the relevant transform code to all Output datasets discovered in 2

5- The CI finishes green, and each dataset can now build the transforms logic wired via the jobspec.

For reference: You can see the jobspec in the details inside the dataset.

user_260040
Related