Duplicate array without doubling peak memory usage in PHP

Viewed 21

I'm looking for the most memory efficient way to process each element in a large PHP array that is already in memory and convert it to another format with minimal peak memory usage increase and without writing data out to disk.

My initial thought was as I loop through the $source_array, I could simply unset() each row to free up memory. However that actually increases the peak memory usage.

Below is example code to showcase the situation:

$source_array = array_fill( 0, 500000, [ 'col1' => 'val1', 'col2' => 'val2', 'col3' => 'val3', 'col4' => 'val4', 'col5' => 'val5',] );
echo "Initial Memory Usage: ". memory_get_usage()  .' Peak Memory Usage: '. memory_get_peak_usage() ."<br>\n";

$destination_array = [];
foreach( $source_array as $key => $row ) {
    $destination_array[$key] = $row; //Assume some manipulation of $row here.

    //$source_array[$key] = null; //Causes peak memory to increase.
    //unset( $source_array[$key] ); //Causes peak memory to increase
    if ( $key > 0 && $key % 100000 == 0 ) {
        echo "  $key. Inside Loop Memory Usage: ". memory_get_usage()  .' Inside Loop Peak Memory Usage: '. memory_get_peak_usage() ."<br>\n"; //Peak memory usage is double initial memory usage.
        gc_collect_cycles();
    }
}
unset( $source_array );
echo "Final Memory Usage: ". memory_get_usage()  .' Final Peak Memory Usage: '. memory_get_peak_usage() ."<br>\n"; //Peak memory usage is double initial memory usage.

Output: NOTICE Initial and Final memory usage are identical, but peak memory is double.

Initial Memory Usage: 17250856 Peak Memory Usage: 17250952
100000. Inside Loop Memory Usage: 21449400 Inside Loop Peak Memory Usage: 21449520
200000. Inside Loop Memory Usage: 25643704 Inside Loop Peak Memory Usage: 25643824
300000. Inside Loop Memory Usage: 34032312 Inside Loop Peak Memory Usage: 34032432
400000. Inside Loop Memory Usage: 34032312 Inside Loop Peak Memory Usage: 34032512
Final Memory Usage: 17250856 Final Peak Memory Usage: 34032512

The best I have been able to come up with so far is using SplFixedArray in the following way:

$source_array = array_fill( 0, 500000, [ 'col1' => 'val1', 'col2' => 'val2', 'col3' => 'val3', 'col4' => 'val4', 'col5' => 'val5',] );
echo "Initial Memory Usage: ". memory_get_usage()  .' Peak Memory Usage: '. memory_get_peak_usage() ."<br>\n";

$source_array = SplFixedArray::fromArray( $source_array );
echo "Pre-Loop Memory Usage: ". memory_get_usage()  .' Pre-Loop Peak Memory Usage: '. memory_get_peak_usage() ."<br>\n";

$destination_array = new SplFixedArray( count( $source_array ) ); //This reduces peak memory by about 20-30%.
foreach( $source_array as $key => $row ) {
    $destination_array[$key] = $row; //Assume some manipulation of $row here.

    //$source_array[$key] = null; //Causes peak memory to increase.
    //unset( $source_array[$key] ); //Causes peak memory to increase
    if ( $key > 0 && $key % 100000 == 0 ) {
        echo "  $key. Inside Loop Memory Usage: ". memory_get_usage()  .' Inside Loop Peak Memory Usage: '. memory_get_peak_usage() ."<br>\n"; //Peak memory usage is double initial memory usage.
        gc_collect_cycles();
    }
}
unset( $source_array );
echo "Final Memory Usage: ". memory_get_usage()  .' Final Peak Memory Usage: '. memory_get_peak_usage() ."<br>\n"; //Peak memory usage is double initial memory usage.

Output:

Initial Memory Usage: 17250920 Peak Memory Usage: 17251016
Pre-Loop Memory Usage: 8473216 Pre-Loop Peak Memory Usage: 25254608
100000. Inside Loop Memory Usage: 16477064 Inside Loop Peak Memory Usage: 25254608
200000. Inside Loop Memory Usage: 16477120 Inside Loop Peak Memory Usage: 25254608
300000. Inside Loop Memory Usage: 16477120 Inside Loop Peak Memory Usage: 25254608
400000. Inside Loop Memory Usage: 16477120 Inside Loop Peak Memory Usage: 25254608
Final Memory Usage: 8473272 Final Peak Memory Usage: 25254608

While a ~30% improvement over the original is decent, I feel as though I should be able to do this with a near zero peak memory increase. Any ideas?

0 Answers
Related