Questions tagged as 'web-scraping'

Tagged questions
0
answers

Transparent data download with application form

I'm trying to put together a script for web scraping, however it's a lot of data and using selenium has not been a good solution. To do all the downloading I would demand at least 12 days. How can I optimize this script? I wonder if having a for...
asked by 18.11.2018 / 19:47
0
answers

GET request blocked for Scrap PHP?

I'm making a GET request, via postman, and it works normally. When in localhost, it works too. But when on air, on a server, it seems that the destination site blocks the request. <?php $curl = curl_init(); curl_setopt_array($curl,...
asked by 27.11.2018 / 15:37
1
answer

Data scraping with jsoup and saving in txt

Hello, how are you guys? I'm trying to learn how to scrape data on my account, and since my English does not help, I'm turning 30. It's basically the following. While executing my code, he lists the athletes of the International Judo Federation,...
asked by 29.10.2018 / 19:09
0
answers

Problem To open multiple links with selenium in python

I have a list of addresses, where I need to click on each element in the list, then click the access button on the screen, but I'm only able to click on the first element ... Follow my code and page code, if you can help me, I've been there f...
asked by 16.10.2018 / 22:30
0
answers

Navigate with Curl after executing login

Good morning, I'm trying, after logging into a website, to extract some information, however, after logging into the site, I need to send a new post to select the data. The problem is that in this second request the screen goes blank. I already...
asked by 04.10.2018 / 17:50
1
answer

How to handle ERR_CONNECTION_TIMED_OUT error in webscraping using a list

I'm learning node.js and got with the help of @Sorack joking with a webscraping. With respect to the code the following happens when the statusCode of the page is equal to 200 the page returns the information and generate it in the file as re...
asked by 02.10.2018 / 22:12
1
answer

How to name each row in a url list

Can I name each line in the url list, to return the nickname I gave it to? Type so the result was this: Prefeitura Municipal de Bocaiúva do Sul | PRONIM TB 518.01.07-013 | Prefeitura Municipal de Matinhos | PRONIM TB 518.01.04-000 | | PR...
asked by 01.10.2018 / 22:25
1
answer

Crawler - how to access several pages

I've put a code on the node to search for the system version and the name of the municipality of a portal, but I'm not able to get it to fetch the information of another municipality from just one. On request I would like it to loop and acces...
asked by 28.09.2018 / 20:29
1
answer

How can I use Scrapy on Anaconda

Hello, I'm having trouble creating a project with Scrapy. I'm studying data science in college and I have to use Scrapy. I'm using the Anaconda. First through the Spider IDE (Anaconda Navigator), I am now trying for the same prompt. The problem...
asked by 18.09.2018 / 02:58
1
answer

I can not do web scraping properly from a Python web site

Well, I was making a code that would check the day of each comic / gif page, and if the day is the same as the current day (in the code I put 14 only because the site does not update the weekend and I I needed to test it somehow), I would downlo...
asked by 15.09.2018 / 23:39