Aggregate multiple (many!) pair of columns (Exce)

Viewed 54

I have table; The table consists of pairs of date and value columns

 Pair               Pair              Pair            Pair           .... ..... ......

enter image description here

What I need is the sum of all values for the same date. The total table has 3146 columns (so 1573 pairs of value and date)!! with up to 186 entries on row level.

Thankfully, the first column contains all possible date values.

Considering the 3146 columns I am not sure how to do that without doing massivle amount of small steps :(

3 Answers

Quick google shows "Number of columns per table 16,384" for powerquery and 16000 for powerBI, so I'm thinking you have to split your input data somehow first, or perhaps this is not the tool for you, maybe AWK

Assuming that works, an M version of what you are looking for. It stacks the columns in groups of 2, then groups and sums them

let Source = Excel.CurrentWorkbook(){[Name="Table4"]}[Content],
Combo = List.Split(Table.ColumnNames(Source),2),
#"Added Custom" =List.Accumulate(
    Combo, 
    #table({"Column1"}, {}),
    (state,current)=> state & Table.Skip(Table.DemoteHeaders(Table.SelectColumns(Source, current)),1)
),
#"Grouped Rows" = Table.Group(#"Added Custom", {"Column1"}, {{"Sum", each List.Sum([Column2]), type number}})
in #"Grouped Rows"

enter image description here

This shows a different method of creating the two column table that you will group by Date and return the Sum. Might be faster than the List.Accumulate method. Certainly worth a try in view of your comment above.

  • Unpivot the original table
  • Add 0-based Index column; then IntegerDivide by 2
  • Group by the IntegerDivide column and extract the Date and Value to separate columns.
  • Then group by date and aggregate by sum
let
    Source = Excel.CurrentWorkbook(){[Name="Table12"]}[Content],

//assuming only columns are Date and Value, this will set the data types for any number of columns
    Types = List.Transform(List.Alternate(Table.ColumnNames(Source),1,1,1), each {_, type date}) &
            List.Transform(List.Alternate(Table.ColumnNames(Source),1,1,0), each {_, type number}),
    #"Changed Type" = Table.TransformColumnTypes(Source,Types),

//Unpivot all columns to create a two column table
//The Value.1 table will alternate the related Date/Value
    #"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Changed Type", {}, "Attribute", "Value.1"),

//add a column to group the pairs of values
//below two lines => a column in sequence of 0,0,1,1,2,2,3,3, ...
    #"Added Index" = Table.AddIndexColumn(#"Unpivoted Other Columns", "Index", 0, 1, Int64.Type),
    #"Inserted Integer-Division" = Table.AddColumn(#"Added Index", "Integer-Division", each Number.IntegerDivide([Index], 2), Int64.Type),
    #"Removed Columns" = Table.RemoveColumns(#"Inserted Integer-Division",{"Index"}),

// Group by the "pairing" sequence,
//  Extract the Date and Value to new columns
// => a 2 column table
    #"Grouped Rows" = Table.Group(#"Removed Columns", {"Integer-Division"}, {
        {"Date", each [Value.1]{0}, type date},
        {"Value", each [Value.1]{1}, type number}}),
    #"Removed Columns1" = Table.RemoveColumns(#"Grouped Rows",{"Integer-Division"}),

//Group by Date and aggregate by Sum
    #"Grouped Rows1" = Table.Group(#"Removed Columns1", {"Date"}, {{"Sum Values", each List.Sum([Value]), type number}}),

//Sort into date order
    #"Sorted Rows" = Table.Sort(#"Grouped Rows1",{{"Date", Order.Ascending}})
in
    #"Sorted Rows"

enter image description here

186 rows * 1573 pairs of columns = 292,578 records.
Assuming not a very old version of Excel, 293k rows is fine, so it can be done with formulae:

Insert five columns to the left, so data starts in F3.
In A3 put zero, in A4 put 1, select the two and drag down to A188.
In A189 put =A3.

In B3 put 0, and drag down to B188.
In B189 put =B3

"Drag"* down A189 and B189 to row 292580

In C3 put =OFFSET($F$3,A3,B3)
In D3 put =OFFSET($F$3,A3,B3+1)
Select those two cells and click on the cross at bottom right to copy them to the end of column B.

Then put Date and Value in A1 and B1, and use a Pivot Table to get totals, averages, or whatever you need.
Any blank cells in the original input do not matter.


  • to "drag" down hundred of thousands of cells:
    Copy A189 and B189
    Goto (F5) A292580
    Paste
    Pin (F8)
    CTRL-up arrow
    Enter

And rather than $F$3 I would name that cell Origin, and use "Origin" in the two Offset formulae, but many people seem to consider that too complicated.

Related