Don't try to buffer everything. Just remember where each segment starts, and use fseek() wisely.
Here I use a fixed array presuming up to 10 segments. You may have to either increase this, or possibly make it dynamic and "growable".
(This is just a "rough cut", but may lead you to a solution.)
EDIT: Clarification: The upper loop indexes the beginning of each 'section', storing the offset to the first data entry of each section.
Using the combination of ftell() and fseek(), the lower loop drives a single input FILE to act like many simultaneous FILE buffers reading single lines from the same file but at different locations. (Analogous to sequentially "popping top item" off multiple stacks until a stack is empty. All stacks presumed to be equally full at the outset.)
size_t offsets[ 10 ] = { 0 };
int nStored = 0;
char buf[ 128 ]; // be generous; don't scrimp.
// first pass, just remember position of 1st number
while( fgets( buf, sizeof buf, infp ) )
if( strncmp( buf, "!!!", 3 ) )
offsets[ nStored++ ] = ftell( infp );
// Now, make 100,000 passes until exhaust (equal sized) sections.
while( true ) {
for( int i = 0; i < nStored; i++ ) {
fseek( infp, offsets[ i ], SEEK_SET );
fgets( buf, sizeof buf, infp ) )
// Been 'chewing through' data lines so far.
if( strncmp( buf, "!!!", 3 ) ) // section boundary?
return; // finished with all equal sized data rows.
*strpbrk( buf, "\n" ) = ','; // replace NL with ','
fprintf( outfp, "%s", buf );
offsets[ i ] = ftell( infp ); // update for next pass
}
fprintf( outfp, "\n" ); // Yeah, trailing comma and null field. Life, eh?
}
If the "sections" between "!!!" markers are NOT the same size, then do not "update" the offset value for short sections... When small sections are exhausted, output a "," to the output file to indicate "empty column". Will need a flag to indicate "no new data found during this sweep; all sections have been exhausted" and that is the clue that the job is done.