Questions tagged as 'scrapy'

Tagged questions
1
answer

Parse Xpath from Int

I have a scrapy running the for to bring the day and the link to something. Ex: t_day = div.xpath('.//a/text()').extract_first() a_day = div.xpath('.//a/@href').extract_first() day = int(t_day) if day > last_day: print(t_day, a_day)...
asked by 04.05.2018 / 20:57
0
answers

Variable xpath saved in Mongo

I have tried in many ways, but I believe that I am going the wrong way. I would like to save the XPATH in Mongodb and import in the item () of the function parse. Is there any way to do it?     
asked by 20.04.2018 / 23:39
0
answers

Parse of two different pages

I have a problem and I can not solve it. I have 2 scripts that make the parse of two different pages. Is it possible for me to create a spider class, start these 2 different urls and create a parse for each url? I created 1 script for each class...
asked by 23.03.2018 / 01:03
0
answers

Scrapy Multiples Requests returning the same information

Hello, I'm trying to extract data, but strangely the return is coming by 2 in 2, ie if I make 6 different requests it returns 6 results but only 3 of them are destined. In the example below the response even though a different request (ncm...
asked by 31.01.2018 / 22:57
0
answers

Problem with Scrapy SplashFormRequest

I'm having a problem with Spash forms requests, using only Scrapy the page loading is done normally, but when I use docker + splash the page does not make the next request. Scrapy: # -*- coding: utf-8 -*- import scrapy class QitestSpider...
asked by 04.02.2018 / 20:23
1
answer

Web Data Extraction with Scrapy

Up .. Next guys ... I am writing a code with Framework scrapy to fetch and extract some data. I'm new to this! The following code would theoretically have to fetch and extract, but it does not extract the data when I run it. I'm using the class...
asked by 05.07.2017 / 04:12
1
answer

"missing scheme" error using Scrapy

When I run my spider, the scrapy returns the following error:    ValueError: Missing scheme in request url h import scrapy class QuotesSpider(scrapy.Spider): name = "Mineracao" def start_requests(self): link = "ht...
asked by 06.05.2017 / 21:04
1
answer

twisted critical unhandled error no scrapy tutorial

I'm new to programming and I'm trying to do the scrapy tutorial link Using python 2.7 and windows 7. When I run the "scrapy crawl dmoz" cmd command, I get the following information as a message: 2015-07-14 16:11:02 [scrapy] INFO: Scrapy 1...
asked by 14.07.2015 / 21:21
1
answer

Write file in excel within the Scrapy project [closed]

I have a Spider that takes the xlsx links, in the Request I call the files and saved in: def save_file(self, response): f = open("teste.xls", "wb") f = write(response.body) f.close() But it returns the error: f = write(resp...
asked by 09.05.2018 / 17:47