Ideas on an effective algorithm to summarize occurances per user in a big data source

Viewed 10

Heyo! (first post, yay!)

I'm having some problems with one of our UiPath RPA Automations. It's an automation that fetches a large data-source in table format, does some work with that and presents the data in a new way. The current soulution works OK as long as the numbers are low -- but as soon as I get any volume it seems to grow exponetially slower.

Input and output:

This is an example of how the data-source looks, it contains instances of produced documents. One row equals one document. The amount of rows ranges from 1000-30000:

UserID DateTime Cathegory1 Cathegory2
user1 2022-01-01 Logistics Invoice
user2 2019-01-01 HR Note
user2 2020-01-01 HR Invoice
user1 2021-01-01 Logistics Note
user3 2022-01-01 Logistics Invoice

The output is a new table where we want a summary per user, based on a cathegory and some datetime-intervals. Under the datetime-intervals we see amount of occurances in that interval.

UserID Cathegory1 Cathegory2 2019 2020 2021 2022
user1 Logistics Invoice 5 10 12 24
user2 HR Invoice 2 13 12 14
user2 HR Note 5 10 12 54
user3 Logistics Invoice 5 155 2 14

Current solution:

The output-table is partly delivered already. I have a table with one row per unique user in the input, aswell as the cathegories in place. So what we have to do is populate the 'date-interval'-columns with amount of occurances within that interval.

The date-intervals are dynamic and comes in as a List.

Current code written in some pseudo style:

var table inputTable
var table outputTable
var list[] dateIntervals

For each row in outputTable
        For each DateTime in dateIntervals
                Filter table inputTable
                        where userid=outputTable.currentRow("userid)
                        AND cathegory=outputTable.currentRow("cathegory)
                        AND within dateInterval

                        outputTable.currentRow("dateInterval) = filteredrawData.count
                        // Remaining rows after filter is amnt within that interval

So yea, this double for-each solution works fine with as long as the input-table is not too big.

I'd gladly take some suggestions on how to improve on this to make it faster and more scaleable.

Thanks in advance :)

0 Answers
Related