I'm developing an application in Laravel , where I have to use Web Scraping techniques to extract data from some sites, and make it available to the user. I have already programmed the Spider-Scripts to mine this data, and they are working perfectly, however, it has a very high cost in terms of time / processing. On average, the queries last around 1min, which is not feasible for the user to wait.
Then I saw that it is necessary to use Cron Jobs , for these scripts to run in Background, at pre-established times (dawn), inserting the values in the database, when the user access the information, they are already properly filtered and easily accessible.
-
I would like to know what precautions I should take for domains which will be owned and suspected of attacking DDOS , and what is the best way to do this extraction? Using Cron Jobs or another service?
-
It will be a Multi Tenancy system, hence hundreds of users, and consequently hundreds of queries.