Good practices when working with file processing

Question

Good practices when working with file processing

Navigation

#1 by (2 votes)
#2 by (0 votes)

2

I have a web application in Asp.Net MVC with C # and I received a new requirement where the goal is to read a text file with thousands of lines, each line containing a set of data that will be used to insert and update the database. data.

My question is what best practices to take in this development. With my little experience, I know that the large amount of information makes the processing very time consuming, reaching more than one hour and eventually causing timeout. I believe that just increasing the timeout is not the best solution.

I also need to present to the user the processing situation, preferably in real time, what would be my options?

c# asp.net-mvc

asked by anonymous 06.02.2016 / 00:21

2 answers

Reading json file via AngularJs service How do I open a window inside a main Frame?

score 2 · Answer 1

With my little experience, I know that the large amount of information makes processing time-consuming, time-consuming, and time-consuming. I believe that just increasing timeout is not the best solution.

But in this case it is. No matter how fast your application is, I find it interesting to have a higher timeout in this case.

I also need to present to the user the processing situation, preferably in real time, what would be my options?

Ajax and a pretty progress bar. I suggest the NProgress.js .

score 0 · Answer 2

Take a look at this Macoratti link tutorial and also at this link link

I made an adaptation of this code and use the Windows OLEDB library itself to read the file. Once the connection is created, I make a select of the file and generate a DataTable in memory.

Once loaded into memory, I use the Bulk to load the Bank.

I have done several, and it has worked well for text files of very varied formats.

Important Tip: To do this, you need to create a Schema.ini file at the same address as the source file. This file contains configuration data so the OLEDB library can read the TXT file.