'system.outofmemoryexception' while opening 300 mega .dbf file

1

I try to open a 300mg .dbf file in my code, but this error appears '$ exception.Message' threw an exception of type 'System.OutOfMemoryException' , a smaller file of 300kb is opened normally, I know the file is too large, but I also know that it is possible to execute files of this size (300mg)

I'm using Windows Form ...

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Data.OleDb;
using System.Collections;
using System.Threading;

namespace LerDBF
{

public partial class Form1 : Form
{
    public Form1()
    {
        InitializeComponent();
    }

    private void btnConsultaPrimaria_Click(object sender, EventArgs e)
    {
        try
        {
            OleDbConnection oConn = new OleDbConnection();
            oConn.ConnectionString = "Provider = Microsoft.Jet.OLEDB.4.0; Data Source= c:\; Extended Properties = dBASE IV; User ID=; Password=";
            OpenFileDialog openFileDialog1 = new OpenFileDialog();
            openFileDialog1.Filter = "Arquivos DBF|*.dbf";
            openFileDialog1.Title = "Selecione o arquivo DBF";

            if (openFileDialog1.ShowDialog() == System.Windows.Forms.DialogResult.OK)
            {
                oConn.Open();
                OleDbCommand oCmd = oConn.CreateCommand();

                oCmd.CommandText = @"SELECT * FROM " + openFileDialog1.FileName;
                DataTable dt = new DataTable();
                dt.Load(oCmd.ExecuteReader());
                ArrayList dadosComboBox = new ArrayList();
                foreach (DataColumn c in dt.Columns)
                {
                    dadosComboBox.Add(c.ColumnName);
                }
                comboBox1.DataSource = dadosComboBox;
                oConn.Close();
                dataGridView1.DataSource = dt;
            }
        }
        catch (Exception exc)
        {
            MessageBox.Show("Erro: " + exc.Message);
        }
    }

    private void btnConsultaSecundaria_Click(object sender, EventArgs e)
    {
        try
        {
            OleDbConnection oConn = new OleDbConnection();
            oConn.ConnectionString = "Provider = Microsoft.Jet.OLEDB.4.0; Data Source= c:\; Extended Properties = dBASE IV; User ID=; Password=";
            OpenFileDialog openFileDialog1 = new OpenFileDialog();
            openFileDialog1.Filter = "Arquivos DBF|*.dbf";
            openFileDialog1.Title = "Selecione o arquivo DBF";

            if (openFileDialog1.ShowDialog() == System.Windows.Forms.DialogResult.OK)
            {
                oConn.Open();
                OleDbCommand oCmd = oConn.CreateCommand();

                oCmd.CommandText = @"SELECT * FROM " + openFileDialog1.FileName;
                DataTable dt = new DataTable();
                dt.Load(oCmd.ExecuteReader());
                ArrayList dadosComboBox = new ArrayList();
                foreach (DataColumn c in dt.Columns)
                {
                    dadosComboBox.Add(c.ColumnName);
                }
                comboBox2.DataSource = dadosComboBox;
                oConn.Close();
                dataGridView2.DataSource = dt;
            }
        }
        catch (Exception exc)
        {
            MessageBox.Show("Erro: " + exc.Message);
        }
    }
}


}

Would I have to use some thread to run this in the background?

    
asked by anonymous 23.10.2015 / 22:15

2 answers

3

The solution is very simple. Do not load the entire file into memory. When giving SELECT put a WHERE clause to get only part of the .dbf records and process in parts. You will probably have to make a loop to repeat the operation until you do what you want with all the data, but trying to put them all in memory will not work. Depending on the goal, you will have to develop a complex logic to handle this.

You can also use a DataView to virtualize the data. I do not know if it works perfectly with .dbf but I think so.

In some cases a DataReader could be a solution.

    
23.10.2015 / 22:27
1

Add the following code in your App.config:

 <configuration>
   <runtime>
     <gcAllowVeryLargeObjects enabled="true" />
   </runtime>
 </configuration>
    
26.10.2015 / 13:56