How to improve file write speed for a ClientDataSet?

Hello, I've had to do something similar, here are some suggestions that have greatly improved the performance in my case.

1 - Change the LogChanges property of ClientDataSet to False (this greatly decreases the time of bulk inserts)

2 - If you are not displaying the ClientDataSet data in some visual control execute ClienteDataset.DisableControls before starting the inserts

3 - If the ClientDataset is not indexed, preferably by using Insert instead of Append , since Append has to place at the end of the dataset the new record inserted

4 - If you have too many fields in Dataset to eliminate the use of FieldByName can help a lot too. If the fields are already created in the design-time dataset, you can access it directly by the object TField (example: ClienteDataset1NOME.AsString )

5 - The ClientDataSet is not good for insertion of many records, if it is indexed the curve in the operation of many inserts is exponential, this can be evidenced through the ASMProfiler tool. One option that helps in performance is to eliminate the indexes before starting these batch operations and restore the indexes at the end of the operation so the cost becomes more linear.

I do not know the details of your system, but 780 lines of a text file per minute seems low to me, unless you're pulling the file from somewhere on the network.

From what I've seen, all comments so far have focused on clientdataset . I think it's worth changing the focus a little bit.

For "small" files, using StringList to read the txt file is very valid. It's usually faster, easier, looks perfect.

But the fact is: Using StringList , you bring to memory your text file. From what I understand, the file you're using is huge, and in cases like this, I realized that by using the file structure, not bringing the whole txt to memory, things flow better.

I tested it on a file with 500,000 lines (Input.txt).

I first tested StringList : (needless to say it did not roll, right?)

SoonafterusingtheformIproposed:

Follow the example for testing:

You usually call the function this way:

procedure TForm1.Button2Click(Sender: TObject);
var
  arquivo : Tstringlist;
  I: Integer;
  linha : string;
begin
  arquivo := TStringList.Create;
  arquivo.LoadFromFile('c:\Entrada.txt');
  for I := 0 to arquivo.Count - 1 do
  begin
    linha := arquivo.Strings[i];
    . . . (tratamento da linha)
  end;
  showmessage('pronto');
  arquivo.Destroy;
end;

Test by doing this:

var
   arquivo : TextFile;
   linha   : string;
begin
  AssignFile(arquivo,'c:\Entrada.txt');
  Reset(arquivo);
  while not eof(arquivo) do
  begin
    readln(arquivo,linha);
    . . . (tratamento da linha)
  end;
  showmessage('pronto');
  CloseFile(arquivo);

I do not know if this will solve your slowness, but maybe consume less memory to help a little.

To open a text file I prefer direct approach, so the file is released immediately after it is read:

  function StreamToString(Arquivo: String): String;
  var
    oMemoryStream: TMemoryStream;
  begin
    Result := '';
    if FileExists(Arquivo) then
    begin
      oMemoryStream := TMemoryStream.Create;
      try
        oMemoryStream.LoadFromFile(Arquivo);
        SetString(Result, PChar(oMemoryStream.memory), oMemoryStream.Size);
      finally
        FreeAndNil(oMemoryStream);
      end;
    end;
  end;

As for the DataSet, is it really necessary? In my view we use only for aesthetic effect in this case, which can be replaced by byte control and not line control. Transferring the contents of the file to a variable and from this variable directly to the bank your process must have a considerable gain. Working with 200MB files with data from a credit card I made the breaks and ran COMMIT every 10,000 records on an Oracle database and recorded something around 60 records per second. Example:

procedure TForm1.BitBtn1Click(Sender: TObject);
type
  TDados = record
    Nome  : String;
    Obs   : String;
  end;

var
  rDados    : TDados;
  sTexto    : String;
  sAuxiliar : String;
  nCount,
  nIndex    : Integer;
  nCommit   : Integer;

  function Grava(Dados: TDados): Boolean;
  begin
    Result := True;
    // processo de gravação
  end;

  function Commit: Boolean;
  begin
    Result := True;
    // processo de COMMIT
  end;

  function StreamToString(Arquivo: String): String;
  var
    oMemoryStream: TMemoryStream;
  begin
    Result := '';
    if FileExists(Arquivo) then
    begin
      oMemoryStream := TMemoryStream.Create;
      try
        oMemoryStream.LoadFromFile(Arquivo);
        SetString(Result, PChar(oMemoryStream.memory), oMemoryStream.Size);
      finally
        FreeAndNil(oMemoryStream);
      end;
    end;
  end;

begin

  sTexto := StreamToString('C:\Arquivo.csv');
  if Trim(sTexto) = EmptyStr then
  begin
    Raise Exception.Create('Arquivo vazio!');
  end
  else
  begin
    for nCount := 1 to Length(sTexto) do
    begin
      if (sTexto[nCount] in [#10, #13]) and (sAuxiliar <> '') then
      begin
        rDados.Obs := sAuxiliar;
        sAuxiliar  := '';
        Grava(rDados);
        if nCommit < 10000 then // Vai depender do banco de dados
          nCommit := nCommit + 1
        else
        begin
          Commit;
          nCommit := 0;
        end;
      end
      else if (sTexto[nCount] = ';') then
      begin
        nIndex := nIndex + 1;
        case nIndex of
          0 : rDados.Nome := sAuxiliar;
          // demais campos com exceção do ultimo;
        end;
        sAuxiliar := '';
      end
      else if not (sTexto[nCount] in [#10, #13]) then
        sAuxiliar := sAuxiliar + sTexto[nCount];
    end;
  end;
end;