Questions tagged as 'scrapy'

Tagged questions
1
answer

Web Crawler with Django view.py

I am doing a simple web crawler, using django 2.0, I want to capture only the "title" class of the news and then render "return render" to a simple html, below my view.py. At the moment I'm using the "Return HttpRensonse". how can I get the data...
asked by 31.05.2018 / 03:08
0
answers

What is the best way to scratch the Datasus site in Python?

The link is this: link I'm trying to send a POST through the requests with a dictionary containing the categories I want, but then the URL is still static. Do you think Selenium would be more suitable for this? Has anyone done anything l...
asked by 15.12.2017 / 17:07
1
answer

Scrapy different pages

But I'm facing a problem. And I ended up getting confused, I decided to return the code in a functional point. # -*- coding: utf-8 -*- # coding: utf-8 import scrapy from mbu2.items import Mbu2Item2 import urlparse from scrapy.http import Reque...
asked by 12.07.2016 / 22:38
2
answers

How to convert CSV to XLSX with python?

How do I convert a .csv file generated by python to .xlsx ? I have two problems: One of them is that I could not figure out how to do this conversion The second is that even if you pass the crawl <nome> -o <nom...
asked by 20.09.2018 / 06:35
1
answer

Capture using Xpath

I'm doing a capture of a site using python (scrapy) and xpath How to capture only 232.990 of the code below? <div class="price-advantages-container"> <div class="price-comparison"> <div itemprop="price" class="pri...
asked by 28.08.2018 / 20:34
1
answer

Web Scraping - convert HTML table into python dict

I'm trying to turn an HTML table into dict @ python, I came across some problems and I ask for help from you. Follow it as far as I could get ... def impl12(url='http://www.geonames.org/countries/', tmout=2): import requests from bs...
asked by 01.08.2017 / 16:59
2
answers

Concatenating multiple lists with Python

Good afternoon! I have a problem and need help, I'm working with 3 distinct lists that should be added to a dictionary, but so I can capture all the values without one overwriting the other, I need to concatenate those 3 lists. I know that us...
asked by 08.08.2018 / 20:14
1
answer

Create a new function with scrapy

I'm starting to learn scrapy and created the following function: import scrapy class ModelSpider(scrapy.Spider): name = "model" start_urls = [ 'http://www.icarros.com/' ] def parse(self, response): with open('brands....
asked by 06.09.2016 / 19:03
1
answer

Pass a URL list to Scrapy function

I have a Python API that gets two arguments (URL and a user-defined word) and provides in the JSON file how many times the specified word appears in the URL. However, I would like to pass a list of URLs. I would also like to make the request...
asked by 28.09.2018 / 16:33
1
answer

Adjusting csv columns with Scrapy

I have a problem, python by default when it generates the csv file separates the columns by comma, however I need the created items to be transformed into the respective columns, but I can not do the same thing, could anyone help me? Here is the...
asked by 17.09.2018 / 06:04