Foundry Scenarios edited data materialized as dataset

user_16272484 0
Viewed 90

Is it possible to materialize the edits made as part of a scenario into a dataset in foundry?

I want for each scenario to write out the primary keys of the objects edited as part of the scenario.

The motivation is that I need to run multiple processes to compute metrics as part of the changed values for each scenario, at a scale and runtime that is not possible to do with Functions.

Edit with details:

The thing is that I am not doing actual edits to the objects for the object type, I don't want to apply it.

I tested out the "Action Log" and it does not seem like this picks up "uncommitted" actions, meaning actions that is just run as part of a scenario. Also, it does not seem to be a link to the scenario it was a part of, even if the changes were committed.

The workflow is that I have Object Type A, and I define multiple scenarios S on a subset of the objects in A.

Each scenario might make something like 50k edits to a subset of A, through multiple Actions backed by a Function.

I save some of the scenarios. Now I am able to load these scenarios and "apply" them on A again in Workshop.

However I need to be able to get all the primary keys, and the edited values of A materialized into a dataset (for each scenario), as I need to run some transformation logic to compute a metric for the change as part of each scenario (at a scale and execution time not possible in Functions).

The Action Log did not seem to help a lot for this. How do I get the "edits" as part of a saved scenario into a dataset?

foundry-scenarios palantir-foundry
2 Answers

Afaik there's no first class way of doing this (yet).

"Applying" a scenario basically means you're submitting the actions queued on the scenario to the ontology. So neither the actions nor the ontology are aware that they came from a scenario.

What I've been doing to achieve what you're working on is using the "Action Log". It's still in Beta so you might need to ask for it to be enabled. It will allow you on each action to define a "log" object to be created that can track the pks of your edited objects per Action.

How I do the action log is:

My Action log has the "timestamp" of the action when they were run. My Scenarios have the "timestamp" of when it was applied.

Since "Applying a Scenario" means -> actually running all actions on Ontology (underlying data) this gets me this structure if I sort everything by timestamp:

  • Action 1
  • Action 2
  • Scenario A applied
  • Action 3
  • Action 4
  • Scenario B applied

this allows you to do a mapping later on of Action 1/2 must come from Scenario A and Action 3+4 come from Scenario B.

EDIT: Apparently you might be able to use the Scenario RID directly in the Action Logs too (which is a recent addition I haven't adopted yet)

This won't allow you tho to compute (in transforms...) anything BEFORE applying a scenario tho

user_192496860

The only logic you can run BEFORE applying will be functions.

Not sure about your exact logic but Function's Custom Aggregations can be very powerful: Docs here

this might not directly let you calculate the Diff but you could use the scenario compare widgets in workshop to compare your aggregation between multiple Scenarios

e.g. you have a function that sums(total profit)

Your Workshop could show:

Current Data:
$10k
Scenario A:
$5k
Scneario B:
$13k

instead of like:

Scenario A:
-$5k
Scenario B:
+$3k
user_192496860
Related