Nutch 2.3.1僅抓取種子URL

我必須抓取所有鏈接（最多）的少量URL。爲此我使用Apache Nutch 2.3.1搭配hadoop和hbase。以下是用於此目的的nutch-site.xml文件。Nutch 2.3.1僅抓取種子URL

<?xml version="1.0"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 

<!-- Put site-specific property overrides in this file. --> 

<configuration> 
<property> 
    <name>http.agent.name</name> 
    <value>crawler</value> 
</property> 
<property> 
    <name>storage.data.store.class</name> 
    <value>org.apache.gora.hbase.store.HBaseStore</value> 
</property> 
<property> 
    <name>plugin.includes</name> 
<value>protocol-httpclient|protocol-http|indexer-solr|urlfilter-regex|parse-(html|tika)|index-(basic|more|urdu)|urlnormalizer-(pass|regex|basic)|scoring-opic</value> 
</property> 
<property> 
<name>parser.character.encoding.default</name> 
<value>utf-8</value> 
</property> 
<property> 
    <name>http.robots.403.allow</name> 
    <value>true</value> 
<property> 
    <name>db.max.outlinks.per.page</name> 
    <value>-1</value> 
</property> 
<property> 
    <name>http.robots.agents</name> 
    <value>crawler,*</value> 
</property> 

<!-- language-identifier plugin properties --> 

<property> 
    <name>lang.ngram.min.length</name> 
    <value>1</value> 
</property> 

<property> 
    <name>lang.ngram.max.length</name> 
    <value>4</value> 
</property> 

<property> 
    <name>lang.analyze.max.length</name> 
    <value>2048</value> 
</property> 

<property> 
    <name>lang.extraction.policy</name> 
    <value>detect,identify</value> 
</property> 

<property> 
    <name>lang.identification.only.certain</name> 
    <value>true</value> 
</property> 

<!-- Language properties ends here --> 
<property> 
     <name>http.timeout</name> 
     <value>20000</value> 
</property> 
<!-- These tags are included as our crawled documents has started to decrease --> 
<property> 
<name>fetcher.max.crawl.delay</name> 
<value>10</value> 
</property> 
<property> 
    <name>generate.max.count</name> 
    <value>10000</value> 
</property> 

<property> 
<name>db.ignore.external.links</name> 
<value>true</value> 
</property> 
</configuration>

當我抓取幾個URL，只有種子URL被取，然後用此消息爬行端

GeneratorJob: Selecting best-scoring urls due for fetch. 
GeneratorJob: starting 
GeneratorJob: filtering: false 
GeneratorJob: normalizing: false 
GeneratorJob: topN: 20 
GeneratorJob: finished at 2017-04-21 16:28:35, time elapsed: 00:00:02 
GeneratorJob: generated batch id: 1492774111-8887 containing 0 URLs 
Generate returned 1 (no new segments created) 
Escaping loop: no more URLs to fetch now

類似的問題陳述here，但它是爲版本1.1和我已實施該解決方案這對我的情況不起作用。

來源

2017-04-21 Shafiq

您是否找到解決此問題的解決方案？ –

您需要按照注入種子後生成Cycle：Generate> Fetch> Parse> UpdateDb。因爲在單次爬行中，您無法獲取所有鏈接，因此您必須多次執行此循環。 –

你能檢查你的conf/regex-urlfilter.txt是否它的URL過濾正則表達式阻止預期的outlinks。

# accept anything else 
+.

當您設置db.ignore.external.links到true，所以Nutch的將不會從不同主機產生出站鏈接。您需要在您的conf/nutch-default.xml中查詢db.ignore.internal.links屬性，無論是否爲false。否則，將不會生成鏈接。

<property> 
    <name>db.ignore.internal.links</name> 
    <value>false</value> 
</property> 
<property> 
    <name>db.ignore.external.links</name> 
    <value>true</value> 
</property> 
<property>

HTH。

來源

2017-04-23 05:51:24

Nutch 2.3.1僅抓取種子URL

回答

相關問題