在Mac上做一個Avro工作在Hadoop上的時髦問題

Question

我在Mac OSX上工作。我遇到以下異常，試圖運行在hadoop集羣上使用Avro文件的hadoop作業。我使用的Java 1.6和我使用的Snappy版本是1.0.4.1這不是最後一個，但我不能改變它，因爲其他部分可能發生的其他問題。

堆棧跟蹤如下：

java.lang.reflect.InvocationTargetException 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317) 
    at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219) 
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44) 
    at org.apache.avro.file.SnappyCodec.decompress(SnappyCodec.java:58) 
    at org.apache.avro.file.DataFileStream$DataBlock.decompressUsing(DataFileStream.java:343) 
    at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:199) 
    at org.apache.avro.mapred.AvroRecordReader.next(AvroRecordReader.java:67) 
    at org.apache.avro.mapred.AvroRecordReader.next(AvroRecordReader.java:34) 
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210) 
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195) 
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) 
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393) 
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) 
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:263) 
Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path 
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1758) 
    at java.lang.Runtime.loadLibrary0(Runtime.java:823) 
    at java.lang.System.loadLibrary(System.java:1045) 
    at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52) 
    ... 18 more 
13/06/28 16:26:25 WARN mapred.LocalJobRunner: job_local_0001 
org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null 
    at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229) 
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44) 
    at org.apache.avro.file.SnappyCodec.decompress(SnappyCodec.java:58) 
    at org.apache.avro.file.DataFileStream$DataBlock.decompressUsing(DataFileStream.java:343) 
    at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:199) 
    at org.apache.avro.mapred.AvroRecordReader.next(AvroRecordReader.java:67) 
    at org.apache.avro.mapred.AvroRecordReader.next(AvroRecordReader.java:34) 
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210) 
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195) 
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) 
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393) 
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) 
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:263) 
13/06/28 16:26:25 INFO mapred.JobClient: map 0% reduce 0% 
13/06/28 16:26:25 INFO mapred.JobClient: Job complete: job_local_0001 
13/06/28 16:26:25 INFO mapred.JobClient: Counters: 0 
13/06/28 16:26:25 INFO mapred.JobClient: Job Failed: NA 
Exception in thread "main" java.io.IOException: Job failed! 
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1323) 
    at net.reunify.socialmedia.NumberOfTweetsPerUser.run(NumberOfTweetsPerUser.java:105) 
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) 
    at net.reunify.socialmedia.NumberOfTweetsPerUser.main(NumberOfTweetsPerUser.java:120) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208) 

Answer 1

如果可能的話你再從1.0.4改變活潑到1.0.5。希望這會解決這個問題。