Best answeres tagged with web-scraping

0

answers

Data scraping, data collection

I have an application and would like to collect this data in real time, something like web scraping. Follow the image of the application and the data that I need to collect. Articles, references and etc. already help.

asked by 08.08.2018 / 15:52

0

answers

How do I extract data into Scalepy Models.py fields?

I want to remove all "Municipalities" from the tag starting at this page. link And then remove information such as: "county name", "mayor", etc. of the pages of each county in the list. Using the shell with the line to line extract all t...

asked by 29.07.2018 / 20:36

0

answers

Jsoup problem when trying to access sites with ssl - JAVA

I studied java a long time ago, I stopped for years and now I missed it, I decided to take some scrap video lessons on youtube but I can not access sites that use ssl, the rest of the java normally accesses and returns what I ask , but even had...

asked by 19.07.2018 / 17:00

0

answers

Avoiding DDOS Alerts Due to Multiple Requests

I'm developing an application in Laravel , where I have to use Web Scraping techniques to extract data from some sites, and make it available to the user. I have already programmed the Spider-Scripts to mine this data, and they are working...

asked by 01.06.2018 / 18:04

0

answers

Using MechanicalSoup to navigate through the selected dropdown after logging

I used MechanicalSoup to log in to a page. Opening the browser, it remained as one page application opening only the dropdowns. With BeautifulSoup I can select which dropdown I want to open, but I can not do it. >>> page.select('a[hre...

asked by 23.05.2018 / 16:08

0

answers

Python, downloading file on a certain day and time

I want to structure a Python program that downloads files (manga) from a particular site once a week. I'm training, I took the web scraping course, but I'm lost on how to make these requests. I took as an example, the code of this page import...

asked by 10.05.2018 / 00:42

0

answers

How to get the next strong value with jquery?

Given that I have the following HTML structure, how could I get the value, after and before Categoria: Padaria & Panificação

asked by 12.05.2018 / 21:33

0

answers

Syntax Error: AcroForm field object is wrong type in Python

I'm developing a python spider that should collect some information and download the PDF. The code seems to be correct as it downloads to the directory of the first PDF and then returns the .json. However, it happens that after the first read...

asked by 09.05.2018 / 14:17

0

answers

Web Scraping with java and Selenium

I'm trying to make web scraping in my school system , I'm using lib Jsoup to do the parser, the problem is when I log in to the page, Jsoup can not control the page, thus making it difficult to login. To solve this problem, I found the possible...

asked by 05.05.2018 / 21:16

1

answer

Parse Xpath from Int

I have a scrapy running the for to bring the day and the link to something. Ex: t_day = div.xpath('.//a/text()').extract_first() a_day = div.xpath('.//a/@href').extract_first() day = int(t_day) if day > last_day: print(t_day, a_day)...

asked by 04.05.2018 / 20:57

Questions tagged as 'web-scraping'