Questions tagged as 'web-crawler'

Tagged questions
0
answers

Download and save an image automatically using Selenium WebDriver

I'm trying to download the Captcha image and save it through a WebDriver. Problem is when you click "Save". Does anyone know how I can do this? staticvoidMain(string[]args){vardriver=newInternetExplorerDriver();driver.Navigate().GoToUrl("http:/...
asked by 24.04.2018 / 05:56
0
answers

Login to the eCac system with C #

I'm doing some research, and I'm logging into the ecac site using my company's certificate ... using Chrome I can log into the system without any problem because chrome asks for the certificate to authenticate the login. I tried to implement a...
asked by 19.02.2018 / 18:17
0
answers

Robots continues to index excluded pages

I work in a company that owns a Magento 1.9 store and in order to optimize the search it organizes we are trying to delete hundreds of pages indexed by google, however, I am facing two difficulties. Shop: lojamastertoys.com 1) The "Categor...
asked by 07.02.2018 / 12:11
0
answers

How is the Indexing of AngularJS routes by googlebot?

I've been researching how google and facebook crawlers work, but I have not found much about how they work to identify information from web pages. Does this lead me to question how it will index the pages of the application in Angular? I foun...
asked by 04.02.2018 / 01:40
1
answer

Help Chrome Puppeteer Paging

I'm doing a WebCrawler, in the following flow Access a search URL I collect the information Executing pagination (here is the error) By clicking the next page, reloads the same (then tried to force a wait), but still does not work at...
asked by 18.01.2018 / 21:27
1
answer

OpenSSL DOMDocument problem

I'm trying to get information from a site using DOMDocument but it's giving an error. DOMDocument::loadHTMLFile(): SSL operation failed with code 1. OpenSSL Error messages: error:14090086:SSL routines:ssl3_get_server_certificate:certi...
asked by 08.12.2017 / 16:36
1
answer

Mechanize with Nokogiri: trying to fetch information in divs

Hello I am setting up a crawler to get product information, for this I am using the mechanize and consequently the nokogiri, I have a URL ( link ) that returns only one product, but I can not hit the regular expression to get the price of tha...
asked by 26.07.2017 / 23:02
0
answers

Crawler for Woocommerce

Good afternoon friends. I am developing a crawler in php that will do scrapping of some urls that I will inform. I'm trying to make it pull values from a dynamic url, but I'm not getting it. Someone could help me. <?php $page_title...
asked by 09.03.2016 / 21:37
0
answers

Make a parser of a page

I'm trying to get information from a page through the url. I'm developing in symfony and using simple_html_dom or Crowler. But I'm not even close to doing what I need. The page I'm accessing is this: Events SESC São Carlos What I want to...
asked by 22.01.2016 / 02:07
1
answer

How to make a web crawler access pages that need authentication? [closed]

I need to develop a web-crowler where he would go to a page (where login is necessary and I have such credentials) and the "robot" would find all links on the page and list somewhere, being a memo or even a txt file. It would be a similar proces...
asked by 24.03.2014 / 19:23