Querying a lot of MySQL and PHP data

1

Personal the code below makes the parser of the data that are in the openvpn log and stores it in the database that by other codes are shown in a page with table. However, there are many records in a log file! There are over 6 thousand (and it will go up with time) and this is taking a while and eventually breaking the bank with "Too Many Connections" and also to show the data in a table I use the following:

$sql = "SELECT idvpn, comName, realAddr, byS, byR, virtAddr, since,    blockstatus, connectedOr FROM vpn";
$query = mysqli_query($conexao, $sql);
return $query;

Except with this code above, it takes almost 2 minutes to display the data in a table (where you have that next page with a plugin called DataTable). Because of numerous requests - I think - I could not even log in anymore and the data started to be inserted all broken in the bank ... I need urgent help, I need to know what I'm doing wrong!

Here is the parser code that breaks each line of the log and takes the information, checks if it already exists in the database, if it does not exist it adds, if it exists it checks if it is in the log and in the database, connected, otherwise it updates offline.

<?php
$con = mysqli_connect("*******", "root", "*******", "*******");
set_time_limit(0);
$pathlog = "log2.txt";

ovpnParser($con, $pathlog);

function checkDiff($con, &$isinlog){
// echo '<script>console.log("**** START CHECKDIFF ****")</script>';
$databArray = array(); // database array
$sql = "SELECT comName FROM vpn";
$query = mysqli_query($con, $sql);
while ($result = mysqli_fetch_array($query)){
array_push($databArray, $result['comName']);
}
foreach($databArray as $newArray){
if(!in_array($newArray, $isinlog)){
  // echo '<script>console.log("database ok, log off...")</script>';
  $sql = "UPDATE vpn SET connectedOr = 'false' WHERE  comName =   '{$newArray}'"; // tem no bd mas não tem no log. ou seja, desconectado.

  $query = mysqli_query($con, $sql);
}
}
// echo '<script>console.log("**** END CHECKDIFF****")</script>'; 
}

function ovpnParser($con, $pathlog){
// echo '<script>console.log("**** START OVPN PARSER****")</script>';
$isinlog = array(); // log array

$inclients = false;
$handle = fopen($pathlog, "r");
$inclients = false;

while(!feof($handle)){
$line = fgets($handle, 4096);
if (substr($line, 0, 11) == "CLIENT_LIST"){
  if (preg_match("/CLIENT_LIST\t{1,}UNDEF(.*)UNDEF/", $line)){
    $inclients = false;
   }
   else{
    $inclients = true;
   }
  }
  if ($inclients){
  preg_match("/CLIENT_LIST(.*)UNDEF/", $line, $conteudo);
  $partes = preg_split("/\t{1,}/", trim($conteudo[1]));
  array_push($isinlog, $partes[0]);

  $sql = "SELECT comName FROM vpn WHERE  comName = '{$partes[0]}'";
  $query = mysqli_query($con, $sql);
  $rows = mysqli_num_rows($query);

  if ($rows == 0){
    // echo '<script>console.log("new register")</script>';
    $sql = "INSERT INTO vpn (comName, realAddr, virtAddr, byR, byS, since, blockstatus, connectedOr)
    VALUES ('{$partes[0]}', '{$partes[1]}', '{$partes[2]}', '{$partes[3]}', '{$partes[4]}', '{$partes[5]}', 'true', 'true')";
    $query = mysqli_query($con, $sql);
    }
  elseif ($rows > 0) {
    // echo '<script>console.log("log and database ok...")</script>';
    $sql = "UPDATE vpn SET byR = '{$partes[3]}', byS = '{$partes[4]}', connectedOr = 'true' WHERE  comName = '{$partes[0]}'";
    $query = mysqli_query($con, $sql);
  }
 }
 $inclients = false;
 }
// echo '<script>console.log("**** END OVPN PARSER ****")</script>';
checkDiff($con, $isinlog);
}
    
asked by anonymous 09.08.2016 / 17:24

2 answers

1

The problem is exactly what you mentioned. Multiple connections.

This happens because you make these connections within loops . Below I have exemplified, with your own code, how to fix this, because you are performing queries and insertions within loops 4 times. It will not be quick to fix everything. But understanding the logic, it makes it much easier.

I took the checkDiff() function to exemplify. I did not want to change your comments so the comments in UPPERCASE are mine.

The logic is simple:

  • I will create a single string, concatenating the values inside the loop, to perform the insertion later.
  • Out of the loop I will connect and insert the data.

See:

function checkDiff($con, &$isinlog){
    // echo '<script>console.log("**** START CHECKDIFF ****")</script>';
    $databArray = array(); // database array
    $sql = "SELECT comName FROM vpn";

    $query = mysqli_query($con, $sql);

    while ($result = mysqli_fetch_array($query)){

        array_push($databArray, $result['comName']);

    }

    // COLOQUEI A PRIMEIRA PARTE DA STRING FORA DO FOREACH

    $sql = "UPDATE vpn SET connectedOr = 'false' WHERE ";

    // AGORA VAMOS CONCATENAR  A STRING PARA FAZER A INSERÇÃO DE UMA VEZ SÓ

    foreach($databArray as $newArray){

        if(!in_array($newArray, $isinlog)){
          // echo '<script>console.log("database ok, log off...")</script>';
          // AQUI É REALIZADA A CONCATENAÇÃO
          $sql .= " comName =   '{$newArray}' OR"; // tem no bd mas não tem no log. ou seja, desconectado.

        }

    }

    // VAMOS RETIRAR O ULTIMO ''OR'' PARA NÃO HAVER BUGS
    $sql = substr($sql, 0, -2);


    // REPARE QUE EU TIREI A QUERY DO FOREACH (AGORA A CONEXÃO SERÁ EXECUTA UMA VEZ)

    $query = mysqli_query($con, $sql);
    // echo '<script>console.log("**** END CHECKDIFF****")</script>'; 
}

On all other connections you will have to do the same thing. Only the strings will be different.

In the first case we use OR to make a single UPDATE . But in the case of INSERT we will use a comma to delimit the values.

Example:

Multiple insert:

INSERT INTO 'usuarios' ('id', 'nome') VALUES (0,'Cesar'),(0, 'Andrei')

See, from now on, you should always create your scripts in this way. In addition to avoiding these problems your code will be much faster.

I hope this helps you.

    
09.08.2016 / 18:28
2

Well, I do not know if I understood your problem well! But I saw two points:

1) The delay to display the logs on the screen! Well I've had problems with rendering many records, because it really crashes! Until rendering 6,000 records in the gift and via JS, it really takes a lot! The alternative is to leave paging in php. I particularly do not do any application without the doctrine, the paging system is great!

Paging with PHP - Pagination with Doctrine

2) The problem of many connections, so I understood your code whenever some action, you already run in the bank. Even within a foreach!

foreach($databArray as $newArray){
 if(!in_array($newArray, $isinlog)){
  // echo '<script>console.log("database ok, log off...")</script>';
  $sql = "UPDATE vpn SET connectedOr = 'false' WHERE  comName =    '{$newArray}'"; // tem no bd mas não tem no log. ou seja, desconectado.
$query = mysqli_query($con, $sql);
}
}

An alternative would be to concatenate the queries and then execute them in a single request:

$sql = '';
foreach($databArray as $newArray){
 if(!in_array($newArray, $isinlog)){
  // echo '<script>console.log("database ok, log off...")</script>';
  $sql .= "UPDATE vpn SET connectedOr = 'false' WHERE  comName =    '{$newArray}'; "; // tem no bd mas não tem no log. ou seja, desconectado.
}
}

and to execute this $ sql, you can do this: mysqli multiple

    
09.08.2016 / 18:36