2016-02-29 144 views
0

出發試圖讓阿帕奇水槽鳴叫在HortonWorks。 (使用教程點參考)阿帕奇水槽不與Twitter流

水槽配置權。這裏是flume.conf

TwitterAgent.sources = Twitter 
TwitterAgent.channels = MemChannel 
TwitterAgent.sinks = HDFS 

TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource 
TwitterAgent.sources.Twitter.channels = MemChannel 
TwitterAgent.sources.Twitter.consumerKey = <1bL3mTHJBheYNye8cE4vHKBZ8> 
TwitterAgent.sources.Twitter.consumerSecret = <iO3f2GxrrRjtF88eA4AH6AHncz4VbmxxT22fHWzuxCLaejoxdD> 
TwitterAgent.sources.Twitter.accessToken = <22976784986-nfj6qEkECeNfs3AeDLDCqtlMOCl9B1iHb8cgIF> 
TwitterAgent.sources.Twitter.accessTokenSecret = <jnNPtmBxlGA8hQq5ZyxjCJLdyiKN97Xa1JTifpmp5BREf> 

TwitterAgent.sources.Twitter.keywords = hadoop, big data, analytics, bigdata, cloudera, data science, data scientiest, business intelligence, mapreduce, data warehouse, data warehousing, mahout, hbase, nosql, newsql, businessintelligence, cloudcomputing 

TwitterAgent.sinks.HDFS.channel = MemChannel 
TwitterAgent.sinks.HDFS.type = hdfs 
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/usr/lib/flume/tweets 
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream 
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text 
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000 
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0 
TwitterAgent.sinks.HDFS.hdfs.rollCount = 1000 

TwitterAgent.channels.MemChannel.type = memory 
TwitterAgent.channels.MemChannel.capacity = 10000 
TwitterAgent.channels.MemChannel.transactionCapacity = 100 

我跑水槽代理後:

bin/flume-ng agent --conf ./conf/ -f conf/twitter.conf Dflume.root.logger=DEBUG,console -n TwitterAgent 

屏幕凍結在這一點上。讓我知道如何繼續?或者我應該等待沙箱長時間? enter image description here

回答

0

試試這個flume.conf

#flume.conf for twitter 

TwitterAgent.sources = Twitter 
TwitterAgent.channels = MemChannel 
TwitterAgent.sinks = HDFS 

TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource 
TwitterAgent.sources.Twitter.channels = MemChannel 
TwitterAgent.sources.Twitter.consumerKey = <required> 
TwitterAgent.sources.Twitter.consumerSecret = <required> 
TwitterAgent.sources.Twitter.accessToken = <required> 
TwitterAgent.sources.Twitter.accessTokenSecret = <required> 
TwitterAgent.sources.Twitter.keywords = hadoop, big data, analytics, bigdata, cloudera, data science, data scientiest, business intelligence, mapreduce, data warehouse, data warehousing, mahout, hbase, nosql, newsql, businessintelligence, cloudcomputing 

TwitterAgent.sinks.HDFS.channel = MemChannel 
TwitterAgent.sinks.HDFS.type = hdfs 
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/usr/lib/flume/tweets 
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream 
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text 
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000 
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0 
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000 

TwitterAgent.channels.MemChannel.type = memory 
TwitterAgent.channels.MemChannel.capacity = 10000 
TwitterAgent.channels.MemChannel.transactionCapacity = 100 

命令來運行水槽劑

bin/flume-ng agent --conf ./conf/ -f conf/twitter.conf Dflume.root.logger=DEBUG,console -n TwitterAgent

如果你有疑問,您可以檢查官方頁面: https://github.com/cloudera/cdh-twitter-example/blob/master/flume-sources/flume.conf

如果您想了解獲取利用水槽星火數據: https://www.dezyre.com//hadoop-tutorial/flume-hadoop-twitter-data-extraction