Can we use .set, .append or .replace command in ADX to combine data of Table A with Table B?

Pratibha Verma 0
Viewed 26

I have Table A which contains History data of 1 month(August 2022) and Table B in which I am doing real time ingestion . I want to ingest the data of Table A in Table B and also check if there is any duplicate data in Table B after combining data of both the tables.Can we use .append command to combine data of these tables without loosing my existing data?

How to use these tags in this code. I am bit confused on using these tags?

.append Table B with(tags='["TagA","TagB"]') <| 
   Table A
   | where timestamp between (datetime(2022-08-01 T09:00:00.000Z) .. datetime(2022-08-31 T11:00:00.000))
   | project  timeseries, value, timestamp, scaled_value
azure-data-explorer kql append join
1 Answers
1.

Regarding set/append/replace, the documentation is quite clear.
You need .append.

Command If table exists If table doesn't exist
.set The command fails The table is created and data is ingested
.append Data is appended to the table The command fails
.set-or-append Data is appended to the table The table is created and data is ingested
.set-or-replace Data replaces the data in the table The table is created and data is ingested
2.

Regarding the tables` merge, it would be better to write a query that compares the two tables and returns only the data that is actually needed to be appended to table B (As opposed to ingest everything and then delete the excess data).

P.S.
If you need help with such a query, please open a new post for that.

3.

You can read all about tags here.
Theoretically, as part of the ingestion process, you can mark your A table data with a drop-by: tag, so it could be identified after the merge into table B.

David דודו Markovitz0
Related