Questions tagged as 'scrapy'

Tagged questions
1
answer

WebDriver Error in Python3.5 AttributeError: can not set attribute

I need to download content from a website. I made a code in python 3.5. When I run it just for a single page the code works fine but when I put it in a loop or function it gives error. The code as function is as follows: from bs4 import Bea...
asked by 20.09.2016 / 19:16
2
answers

How to protect my Scrapyd server from unauthenticated calls?

Let's say I have the following configuration in scrapy.cfg in Scrapyd. [deploy] url = http://example.com/api/scrapyd/ username = user password = secret project = projectX In the Scrapyd documentation it cites the username and password optio...
asked by 09.01.2015 / 21:15
1
answer

Implement queues to manage competition between spiders in Scrapyd

Is there any way for Scrapyd to create queues of spiders so that when I send many spiders (with different functions) I can privilege / limit the competition between them? Today, all the Spiders I send execute in the order set by the Scrapyd serve...
asked by 09.01.2015 / 21:08
1
answer

Scrapy queue queue and mysql store

I've grouped 2 questions because I think they're related. I've done a script test , where saved links are stored in the database with your data. Is this a bad practice? (High Priority) Do I have to do something more to avoid duplica...
asked by 13.07.2016 / 18:53
2
answers

I need help in a python crawler

from scrapy.spiders import BaseSpider from scrapy.selector import HtmlXPathSelector from crawler.items import crawlerlistItem class MySpider(BaseSpider): name = "epoca" allowed_domains = ["epocacosmeticos.com.br"] start_urls = ["http...
asked by 16.02.2017 / 00:32
3
answers

Regular expression in python 3.6 for phrase extraction inteitra

I need to extract only the sentences that contain ADMINISTRATION - JUDGE OF OUTSIDE - NIGHT - SISU - GROUP B, for example. That is, I need to get only the course name, city, shift, O SISU, and the group name of the following string: string = &...
asked by 28.02.2017 / 17:33
1
answer

Problems with parameter restrict_xpaths in a crawler

I have no Python experience but I decided to try doing anything with Scrapy for testing. So I'm trying to collect the existing articles on a given page, namely a DIV element with a devBody ID. In this sense, my goal is to get the title of t...
asked by 10.03.2016 / 20:15
1
answer

pass input when executing bat

Good afternoon, I have a .bat to run a python file, it needs to receive an entry, but I can not run this input when running .bat, does anyone know how to do it? follows bat information: cd C:\Users\Jonathan Pereira\Desktop\ProjetoIruan\Robo...
asked by 26.09.2018 / 19:54
1
answer

Scraping parameters from a post method, with scrapy in python!

I need to collect information from a site using spiders within ScraPy in Python, however the site is a post method and I am learning the language while developing the project. I found a template for post but I'm not able to run it correctl...
asked by 07.05.2018 / 13:34
1
answer

Scrapy 1.0 - Log Settings

I need to know how I can do to change the highlighted fields, because when I run my program with Scrapy in version 1.0 it printed the result in these highlighted quantities. I wanted to know how to change them, whether it's through a sc...
asked by 29.03.2016 / 15:10