使用Artoo js與Google Puppeteer進行網絡抓取

Question

我似乎無法使用Artoo和Puppeteer。我試圖通過npm install artoo-js使用它，它沒有工作。我也嘗試使用木偶命令page.injectFile(filePath)注入構建路徑dist。沒有運氣。任何人都能夠成功地暗示這兩個庫？如果是的話，會喜歡artoo的注入代碼片段。

Answer 1

我只是想爲木偶another answer，我想我可以嘗試Artoo了，所以在這裏你去:)

（第0步：安裝Yarn，如果你沒有的話）

yarn init 
yarn add puppeteer 
# Download latest artoo script, not as a yarn dependency here because it won't be by the Node JS runtime 
wget https://medialab.github.io/artoo/public/dist/artoo-latest.min.js 

在index.js保存此：

const puppeteer = require('puppeteer'); 
(async() => { 
    const url = 'https://news.ycombinator.com/'; 
    const browser = await puppeteer.launch(); 
    const page = await browser.newPage(); 
    // Go to URL and wait for page to load 
    await page.goto(url, {waitUntil: 'networkidle'}); 
    // Inject Artoo into page's JS context 
    await page.injectFile('artoo-latest.min.js'); 
    // Sleeping 2s to let Artoo initialize (I don't have a more elegant solution right now) 
    await new Promise(res => setTimeout(res, 2000)) 
    // Use Artoo from page's JS context 
    const result = await page.evaluate(() => { 
     return artoo.scrape('td.title:nth-child(3)', { 
      title: {sel: 'a'}, 
      url: {sel: 'a', attr: 'href'} 
     }); 
    }); 
    console.log(`Result has ${result.length} items, first one is:`, result[0]); 
    browser.close(); 
})(); 

結果：

$ node index.js 
Result has 30 items, first one is: { title: 'Headless mode in Firefoxdeveloper.mozilla.org', 
url: 'https://developer.mozilla.org/en-US/Firefox/Headless_mode' } 

_{這是太可笑錯過：現在HackerNews頂部文章是關於Firefox的無頭...}