User-defined aggregate in PostgreSQL overwrites internal state, if used twice in SELECT

I have a self-written aggregate in C:

create aggregate avg(double precision[])
(
    sfunc = mat_avg,
    stype = float8[],
    finalfunc = mat_avg_final,
    initcond = '{0.0, 0.0}'
);

This should call mat_avg(a function written by me and imported into psql) for every row, and end on a single call of mat_avg_final. This basically assumes, that every float-array is of the same dimensions, and adds all of them up, whilst keeping track of the number of rows(thats the reason for the two element vector {0.0, 0.0}).

Now, if I call this on a single attribute in my select, it works perfectly fine, yet if I call it twice, or more, the state, that is carried throughout the aggregates computation seems to be overwritten.

mat_avg: first call!
mat_avg: end of step 1
mat_avg: first call!
mat_avg: end of step 1
mat_avg:
     dims -> left: [1,3]
     dims -> right:[1,3]
mat_avg: end of step 2
mat_avg:
     dims -> left: [1,4]
     dims -> right:[1,4]
mat_avg: end of step 2
mat_avg:
     dims -> left: [1,4]
     dims -> right:[1,3]

And as you can see, even weirder, it only happens on the third try, not even the second one. Is there something I am missing, in regards to PostgreSQL C-Userdefined-aggregates? Is there only one state?

Does it have to do with the fact that i am using float8[] as the initial condition, even though all are carried as Datum internally anyways?

EDIT: All float8[] are assumed to be matrices, and I add them up element-wise, to get a element-oriented average, with the result being the same dimensionality as the inputs.

EDIT-2:

create table iris(img float[], one_hot float[]);
insert into iris(
   (select array_agg(array_agg) from generate_series(1,2), (select array_agg(random()) from generate_series(1,3)) as alias1), 
   (select array_agg(array_agg) from generate_series(1,3), (select array_agg(random()) from generate_series(1,4)) as alias1)
);
select avg(img) as img_avg, avg(one_hot) as one_hot_avg from iris;