Mule 4.4 DataWeave append counter to file

user_18652037 0
Viewed 127

I am reading in a file (see below). The example file has 13 rows.

A|doe|chemistry|100|A|
B|shea|maths|90|A|
C|baba|physics|80|B|
D|doe|chemistry|100|A|
E|shea|maths|90|A|
F|baba|physics|80|B|
G|doe|chemistry|100|A|
H|shea|maths|90|A|
I|baba|physics|80|B|
J|doe|chemistry|100|A|
K|shea|maths|90|A|
L|baba|physics|80|B|
M|doe|chemistry|100|A|

Then iterating over these rows using a for each ( batch size 5 ) and then calling a REST API Depending on REST API response ( success or failure ) I am writing payloads to respective success / error files.

I have mocked the called API such that first batch of 5 records will fail and rest of the files will succeed.

While writing to success / error files am using the following transformation :

output application/csv quoteValues=true,header=false,separator="|"
---
payload

All of this works fine.
Success log file:

"F"|"baba"|"physics"|"80"|"B"
"G"|"doe"|"chemistry"|"100"|"A"
"H"|"shea"|"maths"|"90"|"A"
"I"|"baba"|"physics"|"80"|"B"
"J"|"doe"|"chemistry"|"100"|"A"
"K"|"shea"|"maths"|"90"|"A"
"L"|"baba"|"physics"|"80"|"B"
"M"|"doe"|"chemistry"|"100"|"A"

Error log file:

"A"|"doe"|"chemistry"|"100"|"A"
"B"|"shea"|"maths"|"90"|"A"
"C"|"baba"|"physics"|"80"|"B"
"D"|"doe"|"chemistry"|"100"|"A"
"E"|"shea"|"maths"|"90"|"A"

Now what I want to do is append the row/line number to each of these files so when this goes to production , whoever is monitoring these files can easily understand and correlate with the original file . So as an example in case of error log file ( the first batch failed which is rows 1 to 5 ) I want to append these numbers to each of the rows:

"1"|"A"|"doe"|"chemistry"|"100"|"A"
"2"|"B"|"shea"|"maths"|"90"|"A"
"3"|"C"|"baba"|"physics"|"80"|"B"
"4"|"D"|"doe"|"chemistry"|"100"|"A"
"5"|"E"|"shea"|"maths"|"90"|"A"

Not sure what I should write in DataWeave to achieve this?

mule4 dataweave mule
2 Answers

Inside the ForEach scope, you have access to the counter vars.counter (or whatever name you've chosen since it's configurable). You will need to iterate over each chunk of records for adding the position for each one. You can use something like:

%dw 2.0
output application/csv quoteValues=true,header=false,separator="|"
var batchSize = 5
---
payload map ({
    counter: batchSize * (vars.counter - 1) + ($$ + 1)
} ++ $
)

Or if you prefer to use the update function (this will add the record counter at the last column instead though):

%dw 2.0
output application/csv quoteValues=true,header=false,separator="|"
var batchSize = 5
---
payload map (
    $ update {
        case .counter! -> batchSize * (vars.counter - 1) + ($$ + 1)
    }
)

Remember to replace the batchSize variable from this code with the same value you're using in the ForEach scope (if it's parameterised, it would be better).

Edit 1 - Clarification: the - 1 and + 1 are because both indexes (the counter from the For Each scope and the $$ from the map) are zero-based.

user_39932410

Just another workaround and to simplify without using any external variables. The script can be split into two; 1st is for Error group and 2nd is for Success.

%dw 2.0
output application/csv quoteValues=true,header=false,separator="|"
// Will be used for creating a counter for Error group
var errorIdx = 1  
 // Will be used for creating a counter for Success group
var successIdx = 6
---
//errorItems for the first 5 rows
(payload[0 to 4] map (items,idx) -> (({"0":(idx) + errorIdx} ++ items))) 

++

//successItems from 6 and remaining items.
(payload[5 to -1] map (items,idx) -> (({"0":(idx) + successIdx} ++ items)))

DataWeave Inline Variables:

  • errorIdx is a pointer for starting the error counter
  • successIdx is a pointer for starting the success counter

This will extract from index 0 to 4 element:

payload[0 to 4]

This will extract from index 5 to remaining elements:

payload[5 to -1]
user_38691940
Related