How to split a 1D numpy array into chunks, with each chunk length depending on condition

Question

I'm new into python and programming in general, so please enlighten me.

I have two 1D arrays for data, time. For each element in time corresponds an element in data. The elements in time correspond to a full day of measurements.

I want to split the data array into 288 chunks that each corresponds into a 5minute chunks of the time array, then take the mean value of each data chunk.

Time array contains elements in seconds counting from a specific point in the past. The difference of N+1 and N element equals to the time difference between the two data samples.

My goal is to split the time array every 300seconds, then find the corresponding elements in data array. The sampling rate between two data elements is not always the same, sometimes its every 11.3 seconds sometimes its 22.6, so its impossible to split the array instantly with np.split.

This is my code:

matrix = np.zeros((288,1), dtype=float)
mean_timestamps = np.zeros((288,1), dtype=float)
w=0
for i in range(288):
    meanchunk = np.empty((20,1))
    meanchunk[:] = np.nan
    a = w 
    while time[w] - time[a] < 300:     #time difference in seconds
        if w == 4175:
            break
        meanchunk[w-a] = data[w]
        if np.logical_not(np.isnan(data[w])):
            mean_timestamps[i] = time[w]
        else:
            mean_timestamps[i] = np.nan

        matrix[i] = np.nanmean(meanchunk)

    if time[w+1] - time[a] > 300: 
        w = w+1
        break
    else:
        w = w+1
mean_timestamps = mean_timestamps[mean_timestamps !=0]
matrix = matrix[matrix !=0]

The problem is that the resulting arrays "mean_timestamps" and "matrix", have a length of 273 instead of 288.

mean_timestamps length

Edit: Below are some elements of the time array.

1.386280815216000080e+09
1.386280837911999941e+09
1.386280860608999968e+09
1.386280883305999994e+09
1.386280906003000021e+09
1.386280928697999954e+09
1.386280951394999981e+09
1.386280974092000008e+09
1.386280985440000057e+09
1.386280996789000034e+09
1.386281019484999895e+09
1.386281042180999994e+09
1.386281064878000021e+09
1.386281087575000048e+09
1.386281110270999908e+09
1.386281121619999886e+09
1.386281132967999935e+09
1.386281155664000034e+09
1.386281178361000061e+09
1.386281201056999922e+09
1.386281223753999949e+09
1.386281246450999975e+09
1.386281269147000074e+09
1.386281280494999886e+09
1.386281291842999935e+09
1.386281314539999962e+09
1.386281337236999989e+09
1.386281359933000088e+09
1.386281382628999949e+09
1.386281405325999975e+09
1.386281428023000002e+09
1.386281450720000029e+09
1.386281473414999962e+09
1.386281496111999989e+09
1.386281518809000015e+09
1.386281541506000042e+09
1.386281564201999903e+09
1.386281575551000118e+09
1.386281586898000002e+09
1.386281609595000029e+09
1.386281632292000055e+09
1.386281654987999916e+09
1.386281677684999943e+09
1.386281700381000042e+09
1.386281723078000069e+09
1.386281745773999929e+09
1.386281768470999956e+09
1.386281791167999983e+09
1.386281802515000105e+09
1.386281813864000082e+09
1.386281836559999943e+09
1.386281859256999969e+09
1.386281870605000019e+09
1.386281881953999996e+09
1.386281904650000095e+09
1.386281927345999956e+09
1.386281950042999983e+09
1.386281972740000010e+09
1.386281995437000036e+09
1.386282018131999969e+09
1.386282040828999996e+09
1.386282063526000023e+09
1.386282086223000050e+09
1.386282108918999910e+09
1.386282131615000010e+09
1.386282154312000036e+09
1.386282165660000086e+09
1.386282177009000063e+09
1.386282199704999924e+09
1.386282222401999950e+09
1.386282245098000050e+09
1.386282267795000076e+09
1.386282290490999937e+09
1.386282313187999964e+09
1.386282335884999990e+09
1.386282358581000090e+09
1.386282381276999950e+09
1.386282403973999977e+09
1.386282426671000004e+09
1.386282449368000031e+09
1.386282460714999914e+09
1.386282472062999964e+09
1.386282494759999990e+09
1.386282506108000040e+09
1.386282517457000017e+09
1.386282540154000044e+09
1.386282562848999977e+09
1.386282585546000004e+09
1.386282608243000031e+09
1.386282630940000057e+09
1.386282653635999918e+09
1.386282676332000017e+09
1.386282699029000044e+09
1.386282721726000071e+09
1.386282744421999931e+09
1.386282755770999908e+09
1.386282767118999958e+09
1.386282789815000057e+09
1.386282812512000084e+09
1.386282835207999945e+09
1.386282857904999971e+09
1.386282880601999998e+09
1.386282903298000097e+09
1.386282925993999958e+09
1.386282948690999985e+09
1.386282971388000011e+09
1.386282994085000038e+09
1.386283016779999971e+09
1.386283039476999998e+09
1.386283050825000048e+09
1.386283062174000025e+09
1.386283084871000051e+09
1.386283107565999985e+09
1.386283130263000011e+09
1.386283152960000038e+09
1.386283175657000065e+09
1.386283187005000114e+09
1.386283198352999926e+09
1.386283221049000025e+09
1.386283243746000051e+09
1.386283266443000078e+09
1.386283289138999939e+09
1.386283311835999966e+09
1.386283334532000065e+09
1.386283345880000114e+09
1.386283357229000092e+09
1.386283379924999952e+09
1.386283402621999979e+09
1.386283425319000006e+09
1.386283448015000105e+09
1.386283470710999966e+09
1.386283493407999992e+09
1.386283516105000019e+09
1.386283538802000046e+09
1.386283561496999979e+09
1.386283584194000006e+09
1.386283606891000032e+09
1.386283629588000059e+09
1.386283640936000109e+09
1.386283652283999920e+09
1.386283674980000019e+09
1.386283697677000046e+09
1.386283720374000072e+09
1.386283743069999933e+09
1.386283765766000032e+09
1.386283788463000059e+09
1.386283811160000086e+09
1.386283833855999947e+09
1.386283856552999973e+09
1.386283867901000023e+09
1.386283879249000072e+09
1.386283901946000099e+09
1.386283924641999960e+09
1.386283935990999937e+09
1.386283947338999987e+09
1.386283970036000013e+09
1.386283992732000113e+09
1.386284015427999973e+09
1.386284038125000000e+09
1.386284060822000027e+09
1.386284083519000053e+09
1.386284106213999987e+09
1.386284128911000013e+09
1.386284151608000040e+09
1.386284174305000067e+09
1.386284197000999928e+09
1.386284219697000027e+09
1.386284231045000076e+09
1.386284242394000053e+09
1.386284265091000080e+09
1.386284287786999941e+09
1.386284310483000040e+09
1.386284333180000067e+09
1.386284355877000093e+09
1.386284378572999954e+09
1.386284401269999981e+09
1.386284423966000080e+09
1.386284446663000107e+09
1.386284469358999968e+09
1.386284492055999994e+09
1.386284514753000021e+09
1.386284526101000071e+09
1.386284537448999882e+09
1.386284560144999981e+09
1.386284571493999958e+09
1.386284582842000008e+09
1.386284605539000034e+09
1.386284628236000061e+09
1.386284650930999994e+09
1.386284673628000021e+09
1.386284696325000048e+09
1.386284719022000074e+09
1.386284741717999935e+09
1.386284764414000034e+09
1.386284787111000061e+09
1.386284809808000088e+09
1.386284821155999899e+09
1.386284832503999949e+09
1.386284855200999975e+09
1.386284877897000074e+09
1.386284900594000101e+09
1.386284923289999962e+09
1.386284945986999989e+09
1.386284968683000088e+09
1.386284991380000114e+09
1.386285014075999975e+09
1.386285036773000002e+09
1.386285059470000029e+09
1.386285082165999889e+09
1.386285104861999989e+09
1.386285116210999966e+09
1.386285127559000015e+09
1.386285150256000042e+09
1.386285172953000069e+09
1.386285195648000002e+09
1.386285218345000029e+09
1.386285241042000055e+09
1.386285252390000105e+09
1.386285263739000082e+09
1.386285286434999943e+09
1.386285309131000042e+09
1.386285331828000069e+09
1.386285354525000095e+09
1.386285377220999956e+09
1.386285399917999983e+09
1.386285411265000105e+09
1.386285422614000082e+09
1.386285445311000109e+09
1.386285468006000042e+09
1.386285490703000069e+09
1.386285513398999929e+09
1.386285536095999956e+09
1.386285558792000055e+09
1.386285581489000082e+09
1.386285604186000109e+09
1.386285626881999969e+09
1.386285649578000069e+09
1.386285672275000095e+09
1.386285694971999884e+09
1.386285706319999933e+09
1.386285717668999910e+09
1.386285740364000082e+09
1.386285763061000109e+09
1.386285785757999897e+09
1.386285808454999924e+09
1.386285831151000023e+09
1.386285853846999884e+09
1.386285876543999910e+09
1.386285899240999937e+09
1.386285921937000036e+09
1.386285933286000013e+09
1.386285944634000063e+09
1.386285967329999924e+09
1.386285990026999950e+09
1.386286001375000000e+09
1.386286012723000050e+09
1.386286035420000076e+09
1.386286058115999937e+09
1.386286080812999964e+09
1.386286103509000063e+09
1.386286126206000090e+09
1.386286148903000116e+09
1.386286171598999977e+09
1.386286194295000076e+09
1.386286216992000103e+09
1.386286239688999891e+09
1.386286262385999918e+09
1.386286285081000090e+09
1.386286296430000067e+09
1.386286307778000116e+09
1.386286330474999905e+09
1.386286353171999931e+09
1.386286375868000031e+09
1.386286398563999891e+09
1.386286421260999918e+09
1.386286443957999945e+09
1.386286466654000044e+09
1.386286489351000071e+09
1.386286512046999931e+09
1.386286534743999958e+09
1.386286557440000057e+09
1.386286580137000084e+09
1.386286591484999895e+09
1.386286602832999945e+09
1.386286625529999971e+09
1.386286636878000021e+09
1.386286648226000071e+09
1.386286670923000097e+09
1.386286693619999886e+09
1.386286716315999985e+09
1.386286739012000084e+09
1.386286761709000111e+09
1.386286784405999899e+09
1.386286807102999926e+09
1.386286829798000097e+09
1.386286852494999886e+09
1.386286875191999912e+09
1.386286886539999962e+09
1.386286897888999939e+09
1.386286920585000038e+09
1.386286943280999899e+09
1.386286965977999926e+09
1.386286988674999952e+09
1.386287011371000051e+09
1.386287034068000078e+09
1.386287056763999939e+09
1.386287079460999966e+09
1.386287102157000065e+09
1.386287124854000092e+09
1.386287147549999952e+09
1.386287170246999979e+09
1.386287181595000029e+09
1.386287192943000078e+09
1.386287215640000105e+09
1.386287238336999893e+09
1.386287261032999992e+09
1.386287283729000092e+09
1.386287306426000118e+09
1.386287317773999929e+09
1.386287329122999907e+09
1.386287351819999933e+09
1.386287374515000105e+09
1.386287397211999893e+09
1.386287419908999920e+09
1.386287442605999947e+09
1.386287465302000046e+09
1.386287476650000095e+09
1.386287487997999907e+09
1.386287510694999933e+09
1.386287533391999960e+09
1.386287556088000059e+09
1.386287578785000086e+09
1.386287601480999947e+09
1.386287624177999973e+09
1.386287646874000072e+09
1.386287669571000099e+09
1.386287692266999960e+09
1.386287714963999987e+09
1.386287737660000086e+09
1.386287760357000113e+09
1.386287771704999924e+09
1.386287783053999901e+09
1.386287805750000000e+09
1.386287828446000099e+09
1.386287851142999887e+09
1.386287873839999914e+09
1.386287896536999941e+09
1.386287919232000113e+09
1.386287941928999901e+09
1.386287964625999928e+09
1.386287987322999954e+09
1.386287998671000004e+09
1.386288010019000053e+09
1.386288032714999914e+09
1.386288055411999941e+09
1.386288066759999990e+09
1.386288078108999968e+09
1.386288100805000067e+09
1.386288123502000093e+09
1.386288146197999954e+09
1.386288168894999981e+09
1.386288191591000080e+09
1.386288214288000107e+09
1.386288236983999968e+09
1.386288259680999994e+09
1.386288282377000093e+09
1.386288305073999882e+09
1.386288327770999908e+09
1.386288350467000008e+09
1.386288361815000057e+09
1.386288373163000107e+09
1.386288395859999895e+09
1.386288418556999922e+09
1.386288441253999949e+09
1.386288463948999882e+09
1.386288486645999908e+09
1.386288509342999935e+09
1.386288532039999962e+09
1.386288554736000061e+09
1.386288577431999922e+09
1.386288600128999949e+09
1.386288622825999975e+09
1.386288645522000074e+09
1.386288656871000051e+09
1.386288668219000101e+09
1.386288690914999962e+09
1.386288702263000011e+09
1.386288713611999989e+09
1.386288736308000088e+09
1.386288759005000114e+09
1.386288781700999975e+09
1.386288804398000002e+09
1.386288827094000101e+09
1.386288849790999889e+09
1.386288872487999916e+09
1.386288895184000015e+09
1.386288917880000114e+09
1.386288940576999903e+09
1.386288951924999952e+09
1.386288963273999929e+09
1.386288985970999956e+09
1.386289008665999889e+09
1.386289031362999916e+09
1.386289054059999943e+09
1.386289076756999969e+09
1.386289099453000069e+09
1.386289122148999929e+09
1.386289144845999956e+09
1.386289167542999983e+09
1.386289190239000082e+09
1.386289212936000109e+09
1.386289235631999969e+09
1.386289246980000019e+09
1.386289258328999996e+09
1.386289281025000095e+09
1.386289303721999884e+09
1.386289326417999983e+09
1.386289349115000010e+09
1.386289371811000109e+09
1.386289383160000086e+09
1.386289394507999897e+09
1.386289417204999924e+09
1.386289439901000023e+09
1.386289462596999884e+09
1.386289485293999910e+09
1.386289507990999937e+09
1.386289530687999964e+09
1.386289542035000086e+09
1.386289553382999897e+09
1.386289576079999924e+09
1.386289598776999950e+09
1.386289621473999977e+09
1.386289644170000076e+09
1.386289666865999937e+09
1.386289689562999964e+09
1.386289712259999990e+09
1.386289734956000090e+09
1.386289757653000116e+09
1.386289780348999977e+09
1.386289803046000004e+09
1.386289825742000103e+09
1.386289837091000080e+09
1.386289848438999891e+09
1.386289871134999990e+09
1.386289893832000017e+09
1.386289916528000116e+09
1.386289939224999905e+09
1.386289961921999931e+09
1.386289984618000031e+09
1.386290007313999891e+09
1.386290030010999918e+09
1.386290052707999945e+09
1.386290075404999971e+09
1.386290098099999905e+09
1.386290120796999931e+09
1.386290132144999981e+09
1.386290143493999958e+09
1.386290166190999985e+09
1.386290188887000084e+09
1.386290211582999945e+09
1.386290234279999971e+09
1.386290256976999998e+09
1.386290279673000097e+09
1.386290302369999886e+09
1.386290325065999985e+09
1.386290347763000011e+09
1.386290370459000111e+09
1.386290393155999899e+09
1.386290415851999998e+09
1.386290427200000048e+09
1.386290438549000025e+09
1.386290461244999886e+09
1.386290483941999912e+09
1.386290506638999939e+09
1.386290529335000038e+09
1.386290552030999899e+09
1.386290574727999926e+09
1.386290597424999952e+09
1.386290620121999979e+09
1.386290642816999912e+09
1.386290665513999939e+09
1.386290688210999966e+09
1.386290710907999992e+09
1.386290722256000042e+09
1.386290733604000092e+09

user_2823755 · Answer

Edit:

This is closer: find the end times for each 300 second chunk beginning with the first value of the array then use boolean indexing to extract chunks and get the means. Still using regular python for loops.

span = a[-1] - a[0]
nbr_of_intervals = int((span//300) + 1)
ends = (np.zeros(nbr_of_intervals)+300).cumsum() + a[0]
averages = []
begin = a[0]
for end in ends:
    chunk = a[(begin <= a) & (a < end)]
    averages.append(chunk.mean())
    begin = end

Throw in some broadcasting.

q = a[:,None] >= ends    # shape (485,34) using the array values from the question.
r = q.argmax(axis=0)
r = r[r.nonzero()]       # shape (33,) - indices to split on
chunks = np.split(a,r)
avgs = [ary.mean() for ary in chunks]

How to split a 1D numpy array into chunks, with each chunk length depending on condition

1 Answers