紅寶石機械化刮ResponseCodeError

Question

我想刮網站的所有搜索結果頁，它的工作原理，但有時腳本將停止，並顯示以下錯誤：

502 => Net::HTTPBadGateway for https://website.com/id/12/ -- unhandled response (Mechanize::ResponseCodeError) 

我想繼續即使它找到腳本一個錯誤。

我的腳本：

require 'mechanize' 
require 'csv' 

a = Mechanize.new 
CSV.open('datas.csv', "wb") do |csv| 
    page = a.get("https://website.com/?page=1-200") #498 
    number = 0 
    page.links_with(:class => "btn btn-default").each do |link| 
     post_link = link.href 
     inside_page = a.get("https://website.com#{post_link}") 
     title = inside_page.at("h1.serviceTitle").text.strip 
     author = inside_page.at(".name").text.strip 
     number+=1 
     csv << [title, author] 
    end 
end 

任何想法？

Answer 1

這很容易通過適當的異常處理來解決。你可以check this page for a better explanation

爲你的代碼，你可以處理該異常，像這樣

CSV.open('datas.csv', "wb") do |csv| 
    begin 
    a = Mechanize.new 
    page = a.get("https://website.com/?page=1-200") #498 
    number = 0 
    page.links_with(:class => "btn btn-default").each do |link| 
     post_link = link.href 
     inside_page = a.get("https://website.com#{post_link}") 
     title = inside_page.at("h1.serviceTitle").text.strip 
     author = inside_page.at(".name").text.strip 
     number+=1 
     csv << [title, author] 
    end 
    rescue => e 
    // do nothing and move on to the next line 
    end 
end