Help Chrome Puppeteer Paging

0

I'm doing a WebCrawler, in the following flow

  • Access a search URL
  • I collect the information
  • Executing pagination (here is the error)
  • By clicking the next page, reloads the same (then tried to force a wait), but still does not work at all.

    Here's a snippet

      

    Puppter.ts

    export class PuppeteerChrome {
    public browser : Promise;
    public page : EventEmitter;
    
    constructor() {
    }
    
    async initializeBrowser() {
        this.browser = await puppeteer.launch({ headless: false, ignoreHTTPSErrors: true, args: ['--no-sandbox', '--disable-setuid-sandbox']});
        this.page = await this.browser.newPage();
        this.page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36')
        this.page.setViewport({
            width: 1920,
            height: 1080
        });   
    }
    
    async navigateTo(url) {
        if (!url)
          throw new Error('Missing URL');
          return await this.page.goto(url, {
            waitUntil: 'networkidle',
            timeout: 180000
        });
    }
    
    destroy() {
        if (this.browser)
          this.browser.close();
    }
    
    async closeBrowser() {
        return await this.browser.close();
    }
    
    async pages() {
        return await this.browser.pages();
    }
    
    async executeScript(jQueryExpression) {
        try {
          const scriptResult = await this.page.evaluate(jQueryExpression);
          return scriptResult;
        } catch (err) {
          console.error(err);
          return {};
        }
    }
    
    async waitPageLoad() {
        let pageready =  false;
        do {
        let readyState = await this.executeScript('document.readyState;');
        pageready = readyState == 'complete' || readyState == 'interactive';
        } while (!pageready)
        return pageready;
    } }
    
      

    client.ts

    //browser
    this.chrome = new PuppeteerChrome ();
    
    // Next Page
    await this.chrome.executeScript(scripts["ScriptNextPageBrowser"]);
    
    // tentztiva 2
    this.chrome.executeScript(scripts["ScriptNextPageBrowser"]).then(() => {
       this.chrome.waitPageLoad();
    });
    

    Does anyone have any tips?

        
    asked by anonymous 18.01.2018 / 21:27

    1 answer

    0

    I ended up resolving with a waitforTimeout() after clicking.

        
    22.01.2018 / 17:08