arrays - Allowed memory size exhausted in PHP for loop

I'm facing a fatal error while I'm trying to manipulate a huge array of arrays in PHP and return the result as a response of an HTTP POST request:

Allowed memory size of 536870912 bytes exhausted

I have already tried to set ini_set('memory_limit', '-1'); in order to see if I get the result, but I didn't get any type of response. Postman crashed all the times that I tried to make the POST request.

The starting structure of the array is this one. The body size is around 25mb. The main array contains around 22k arrays with this structure, I have just included 2:

Array
(
    [0] => Array

        (
            [id] => 14
            [isActive] => 1
            [personId] => 0023fff16d353d16a052a267811af53bc8bd42f51f2266a2904ca41db19dfd32_0
            [gender] => m
            [age] => 69
            [linedata_0] => 2018-03-01 17:15:18, 155.59, 294.076; 2018-03-01 17:16:04, 502.968, 249.947; 2018-03-01 17:16:44, 276.837, 270.593; 2018-03-01 17:17:28, 431.68, 371.14; 2018-03-01 17:17:34, 851.622, 355.915
        )

    [1] => Array

        (
            [id] => 180
            [isActive] => 1
            [personId] => 02659982ae8286409cc5bb283089871b62f2bafbbad517941d64e77ecf2b62b1_0
            [gender] => m
            [age] => 69
            [linedata_0] => 2018-03-01 13:20:05, 155.599, 293.841; 2018-03-01 13:20:48, 495.468, 249.582; 2018-03-01 13:21:28, 258.791, 260.748; 2018-03-01 13:23:20, 859.061, 352.237; 2018-03-01 13:23:32, 56.1404, 269.858
        )
)

Here below the php part for manipulate the array in order to have the expected final result by explode the timestamp and coordinates for each user:

$final_result = [];

foreach($query_result as $row)
{
    $line_datas =explode(";",$row["linedata_0"]);
    $linedata = [];
    $final = [];

    $d = [];

    for($s =0; $s < count($line_datas); $s++){
        $line_data = explode(",",$line_datas[$s]);
        $d["timestamp"] = utf8_encode($line_data[0]);
        $d["x"]= utf8_encode($line_data[1]);
        $d["y"] = utf8_encode($line_data[2]);

        array_push($linedata,$d);
    }


    $final["id"]= $row["id"];
    $final["isActive"]= $row["isActive"];
    $final["personId"]= utf8_encode($row["personId"]);
    $final["name"] = NULL;
    $final["gender"] = utf8_encode($row["gender"]);
    $final["age"] = utf8_encode($row["age"]);
    $final["linedata"]=$linedata;

    array_push($final_result, $final);

}

return $final_result;

As it seems to me there are no infinite loop or bad practices that can justify a memory issue. The only real problem could be the size of the array that need to be manipulated.

Any suggestions?

Answer

This answer is an example of how to implement a buffer(a limited array in memory) in your code and when it is filled, flush it's contents to disk, at the end you will find a huge array on disk in JSON format. I used this way in a situation similar to yours and got great result regarding "memory usage", but as I told you in comments you need to rethink why you need that HUGE array in the first place, and if there is a way to avoid it, go with it.

using this function will save you the memory used by your $final_result array and replace it with $final_result string buffer but we are controlling it's use of memory. However your $query_result array will still taking the memory it needs.

Note that you need to alter the function as you need because I used your variables which are undefined in my code.

/**
 * proccess the HUGE array and save it to disk in json format [element,element]
 * 
 * @param string $fileName absulote file name path you want to save the proccessed array in
 * @return int processed elements count

 */
function buildMyArrayInFile($fileName)
{
    $flushCheckPoint = 100;// set the buffer size as needed, depending on the size of PHP allowed memory and average element size
    $processedElements = 0;
    $final_result = "[";

    file_put_contents($fileName, "");//prepare the file and erase anything in it

    foreach($query_result as $row)

    {
        $line_datas =explode(";",$row["linedata_0"]);
        $linedata = [];
        $final = [];
        $d = [];

        for($s =0; $s < count($line_datas); $s++){
            $line_data = explode(",",$line_datas[$s]);
            $d["timestamp"] = utf8_encode($line_data[0]);
            $d["x"]= utf8_encode($line_data[1]);

            $d["y"] = utf8_encode($line_data[2]);

            array_push($linedata,$d);
        }

        $final["id"]= $row["id"];
        $final["isActive"]= $row["isActive"];
        $final["personId"]= utf8_encode($row["personId"]);
        $final["name"] = NULL;
        $final["gender"] = utf8_encode($row["gender"]);

        $final["age"] = utf8_encode($row["age"]);
        $final["linedata"]=$linedata;


        $final_result .= json_encode($final) . ",";
        $processedElements ++;
        if($processedElements % $flushCheckPoint === 0){
            //the array has reached the limit, flush the array to disk
            file_put_contents($fileName, $final_result, FILE_APPEND);
            $final_result = "";

        }

    }

    $final_result = rtrim($final_result, ",");//trim the last comma
    $final_result .= "]";
    //flush the remaning data in $final_result
    file_put_contents($fileName, $final_result, FILE_APPEND);

    return $processedElements;


}

this is another simple version of the function for testing

// test
var_dump(buildMyArrayInFile2("/home/myuser/myArray.json"));
// outputs int(7)




function buildMyArrayInFile2($fileName)
{
    $flushCheckPoint = 2;// set the buffer size as needed, depending on the size of PHP allowed memory and average element size
    $processedElements = 0;
    $final_result = "[";

    file_put_contents($fileName, "");//prepare the file and erase anything in it


    $test_result = [1,2,3,4,"wee","hello\nworld",5];
    foreach($test_result as $row)
    {
        $final_result .= json_encode($row) . ",";
        $processedElements ++;
        if($processedElements % $flushCheckPoint === 0){
            //the array has reached the limit, flush the array to disk
            file_put_contents($fileName, $final_result, FILE_APPEND);
            $final_result = "";
        }


    }

    $final_result = rtrim($final_result, ",");//trim the last comma
    $final_result .= "]";
    //flush the remaning data in $final_result
    file_put_contents($fileName, $final_result, FILE_APPEND);

    return $processedElements;
}