Questions tagged as 'scrapy'

Tagged questions
1
answer

Fix Encoding problem when exporting to csv from a scrapy file

How can I fix the encoding problem when saving the file to csv? this problem is only happening when saved in csv. fromscrapyimport*fromprojeto_iruan.itemsimport*importcsvclassimprensaNacional(scrapy.Spider):name='imprensaNacional'start_urls=['h...
asked by 18.09.2018 / 03:27
1
answer

Scrapy for login

I got this code from the internet and I changed a little, to log in to the cpfl website, but when I use the command scrapt crawl myproject nothing happens and the command scrapy runspider items.py gives the error error:    No el...
asked by 08.08.2018 / 15:44
1
answer

Save Excel file in Python through Scrapy

How do I make my spider save in a single XML file all the Excel data from the links that I extract? Or do you also save in each single XLS file in the project folder? Part of my spider: def parse(self, response): divs = response.xpa...
asked by 14.05.2018 / 20:53
1
answer

How do I integrate my Django project with Scrapy? [closed]

I want to develop a simple project using Django where I will create a web page and this page will capture data from other pages. The problem is that I can not integrate Scrapy with Django .     
asked by 26.04.2017 / 14:28
0
answers

Send multiple parameters in Scrapy formdata

Hello, I'm using Scrapy to automate a "query" process on a website. After logging into a site, using FormRequest , I need to make a new request to send several process numbers to the site. I'm currently doing this: d...
asked by 15.10.2018 / 14:24
1
answer

How can I use Scrapy on Anaconda

Hello, I'm having trouble creating a project with Scrapy. I'm studying data science in college and I have to use Scrapy. I'm using the Anaconda. First through the Spider IDE (Anaconda Navigator), I am now trying for the same prompt. The problem...
asked by 18.09.2018 / 02:58
0
answers

How do I extract data into Scalepy Models.py fields?

I want to remove all "Municipalities" from the tag starting at this page. link And then remove information such as: "county name", "mayor", etc. of the pages of each county in the list. Using the shell with the line to line extract all t...
asked by 29.07.2018 / 20:36
0
answers

Move to url with an increment in address

I need to do a crawler on a book site. I can get the data I need from a page and I also got the entire domain, but I wanted to do it in a more orderly and logical way. I'd like to start by extracting for a URL and letting the code go through...
asked by 29.06.2018 / 01:08
0
answers

Syntax Error: AcroForm field object is wrong type in Python

I'm developing a python spider that should collect some information and download the PDF. The code seems to be correct as it downloads to the directory of the first PDF and then returns the .json. However, it happens that after the first read...
asked by 09.05.2018 / 14:17
2
answers

Scrapy xpath href or span within div

Hello, I'm trying to make a scratch where I have to get a link and a text, but I'm having a hard time because of page variations. I have 3 possible variations: 1. <div> <strong> <span style="font-family: arial, helvetica,...
asked by 06.05.2018 / 00:04