Questions tagged as 'web-scraping'

Tagged questions
1
answer

Request API with JavaScript

I'm doing a web application in which the purpose will be to use an API to just list some information (GET) and for that I would use only javaScript and html. The API is this: link and has key (headers). const app = document.getElement...
asked by 11.12.2018 / 12:48
1
answer

Doubt how to shave data like Python using BeautifulSoup Table

I'm a beginner and I'm trying to get a table from the transparency portal site, but I'm not able to get the tag with no data at all. When I open the developer tool I visualize the data that I want the states and the transfer value to be, but w...
asked by 10.11.2018 / 10:18
1
answer

Adjusting csv columns with Scrapy

I have a problem, python by default when it generates the csv file separates the columns by comma, however I need the created items to be transformed into the respective columns, but I can not do the same thing, could anyone help me? Here is the...
asked by 17.09.2018 / 06:04
1
answer

Fix Encoding problem when exporting to csv from a scrapy file

How can I fix the encoding problem when saving the file to csv? this problem is only happening when saved in csv. fromscrapyimport*fromprojeto_iruan.itemsimport*importcsvclassimprensaNacional(scrapy.Spider):name='imprensaNacional'start_urls=['h...
asked by 18.09.2018 / 03:27
1
answer

Specific section break in JSON file with python

Is it possible to perform a line break from a specific JSon thread, transform into an array, and then dynamize? Why I'm asking this .. I'm developing a file mining bot and I came across a situation where some pages return only one file on that l...
asked by 29.05.2018 / 14:01
1
answer

Save Excel file in Python through Scrapy

How do I make my spider save in a single XML file all the Excel data from the links that I extract? Or do you also save in each single XLS file in the project folder? Part of my spider: def parse(self, response): divs = response.xpa...
asked by 14.05.2018 / 20:53
1
answer

Iterating web pages using Requests and Python

I am a beginner in web scraping. How to learn how to make a database from data on selling new cars on some websites. One of the sites eh esse url = https://www.seminovosunidas.com.br/veiculos/page:1?utm_source=afilio&utm_medium=display&...
asked by 19.05.2018 / 07:32
1
answer

How to avoid error Max retries exceeded in scraping in Python?

In Python 3 I made a program to wipe table rows from a multipage public site (97893). And I create a list with the rows of each column and I put sleep to try to avoid that the scraping stops, but even using several times is not working...
asked by 19.01.2018 / 12:41
1
answer

Select an option from the drop down menu Selenium Python

I have a menu that has several options, I want to select only the one that is active. When I give 'variable'.find_element_by_id (' key ') Selenium returns me ALL options. Does the active option have a "selected" marker, as I point to this seleni...
asked by 13.07.2017 / 19:52
1
answer

Access Tag via beautifulsoup

Hello, I'm having trouble accessing the price that is in the third line of the code via beautifulsoup. Does anyone have any idea how to access?                                               <span id="ctl00_Conteudo_ctl01_spanPrecoP...
asked by 05.12.2018 / 14:49