insert in batches using pdo and transaction

1

Good morning guys, I'm kind of new to the world of php mysql and I'm in need of a help

I am inserting data into my bd into 6 tables, the data comes from txt files

Some files have 5000 rows with 163 records separated by;

That's fine, I can already insert the data into the tables

However inserts of files with more than 500 lines are delaying mto

I read and mum site that does the inserts in blocks of hundred in a hundred 400 in 400

using transaction so I did not really understand the concept of the block division I'm doing

with online transaction by line

follows the scenario absented and a piece of code to see if it helps

tab1

tab2 - has foreign key for tab1

tab3 - has foreign key for tab2

tab4 - has foreign key for tab3

tab5 - has foreign key for tab3

tab6 - has foreign key for tab3

while ( ! feof ( $f ) ) {

// Ler uma linha do arquivo
$linha = fgetcsv ( $f, 0, $delimitador );
if (! $linha) {
    continue;
}



    $conn = $calculo->Connect ();
    $conn->beginTransaction ();

    $stmtOrgao = $conn->prepare("INSERT INTO orgao () values(null, ?,?)");
    $dadosOrgao = array (
            $linha [0],
            $linha [1]
    );
    $stmtOrgao->execute ( $dadosOrgao );
    $idOrgao = $conn->lastInsertId ();

    $stmtFuncionario = $conn->prepare("insert into funcionario() values(null,?,?,?,?,?,?,?,?,?,?,?,?,
                                                                             ?,?,?,?,?,?,?,?,?,?,?,?,
                                                                             ?,?,?,?,?,?,?,?,?,?,?,?,
                                                                             ?,?,?,?,?,?,?,?,?,?,?,?,?,?)");

    $dadosFuncionario = array(
        $linha[2],$linha[3],$linha[4],$linha[5],$linha[6],$linha[7],$linha[8],$linha[9],$linha[10],$linha[11],$linha[12],
        $linha[13],$linha[14],$linha[15],$linha[16],$linha[17],$linha[18],$linha[19],$linha[20],$linha[21],$linha[22],
        $linha[23],$linha[24],$linha[25],$linha[26],$linha[27],$linha[28],$linha[29],$linha[30],$linha[31],$linha[32],
        $linha[33],$linha[34],$linha[35],$linha[36],$linha[37],$linha[38],$linha[39],$linha[40],$linha[41],$linha[42],
        $linha[43],$linha[44],$linha[45],$linha[46],$linha[47],$linha[48],$linha[49],$linha[50],$idOrgao);                    

    $stmtFuncionario->execute($dadosFuncionario);
    $idFuncionario = $conn->lastInsertId();

    $stmtFolha = $conn->prepare("insert into folha () values(null,?,?,?,?,?,?,?,?,?,?,?,?,
                                                                  ?,?,?,?,?,?,?,?,?,?,?,?)");



    if(preg_match("/[R$]/i", $linha[67]))
    {

        $separar = explode(" ", $linha[67]);
        $lin = $separar[1];
    }
    else
    {

        $lin = $linha[67];
    }



    $dadosFolha = array(
        $linha[51],$linha[52],$linha[53],$linha[54],$linha[55],$linha[56],$linha[57],$linha[58],$linha[59],$linha[60],$linha[61],
        $linha[62],$linha[63],$linha[64],$linha[65],$linha[66],$lin,$linha[68],$linha[117],$linha[118],$linha[119],$linha[120],
        $linha[121],$idFuncionario);

    $stmtFolha->execute($dadosFolha);
    $idFolha =  $conn->lastInsertId();

    $stmtProvento = $conn->prepare("insert into proventos () values(null, ?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,
                                                                          ?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,
                                                                          ?,?,?,?,?,?,?,?)");

    $dadosProvento = array(
        $linha[69],$linha[70],$linha[71],$linha[72],$linha[73],$linha[74],$linha[75],$linha[76],$linha[77],$linha[78],$linha[79],
        $linha[80],$linha[81],$linha[82],$linha[83],$linha[84],$linha[85],$linha[86],$linha[87],$linha[88],$linha[97],$linha[98],
        $linha[99],$linha[100],$linha[101],$linha[102],$linha[103],$linha[106],$linha[107],$linha[108],$linha[109],$linha[110],
        $linha[111],$linha[112],$linha[113],$linha[114],$linha[116],$idFolha);
    $stmtProvento->execute($dadosProvento);

    $stmtDesconto = $conn->prepare("insert into descontos () values(null, ?,?,?,?,?,?,?,?,?,?,?,?)");
    $dadosDesconto= array(
        $linha[89],$linha[90],$linha[91],$linha[92],$linha[93],$linha[94],
        $linha[95],$linha[96],$linha[104],$linha[105],$linha[115],$idFolha);
    $stmtDesconto->execute($dadosDesconto);

    $stmtOutros = $conn->prepare("insert into outros () values(null, ?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,
                                                                     ?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,
                                                                     ?,?)");

    $dadosOutros = array(
        $linha[122],$linha[123],$linha[124],$linha[125],$linha[126],$linha[127],$linha[128],$linha[129],$linha[130],$linha[131],
        $linha[132],$linha[133],$linha[134],$linha[135],$linha[136],$linha[137],$linha[138],$linha[139],$linha[140],$linha[141],
        $linha[142],$linha[143],$linha[144],$linha[145],$linha[146],$linha[147],$linha[148],$linha[149],$linha[150],$linha[151],
        $linha[152],$linha[153],$linha[154],$linha[155],$linha[156],$linha[157],$linha[158],$linha[159],$linha[160],$linha[161],
        $linha[162],$idFolha);

   $stmtOutros->execute($dadosOutros);

    $conn->commit();
    $conn=null;
}
    
asked by anonymous 22.08.2014 / 16:06

1 answer

2

In case of your code I recommend moving the prepare codes before the while. This way you can go giving insert in the bank with each reading of the file, avoiding to have a lot of variables in memory.

When you do a prepare, you ask the database to translate the query, mount the execution plan, and keep it in the internal cache of the database.

In order to gain the desired performance, you should prepare the querys only once with the prepare method and then just call the bind method of each PDOStatement and the execute method.

You can also use the named bind placeholders method so that it does not have to be everything with queries. Example:

$ stmtFunctional = $ conn-> prepare ("insert into official () values (null,: field1,: field2,: field3)");

Then when you bind in the parameters you can do: $ stmtFuntionary-> bind (': field1', $ field_value);

    
04.09.2014 / 14:42