2017-10-10 619 views
2

我有帶spark(1.6.1),hdfs和hive(2.1)的紗線簇。直到今天,我的工作流程都可以正常工作幾個月(沒有任何代碼/環境變化)。我開始像這樣得到錯誤:org.apache.hive.com.esotericsoftware.kryo.KryoException:遇到未註冊的類ID:21

org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 21 
Serialization trace: 
outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc) 
aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork) 
invertedWorkGraph (org.apache.hadoop.hive.ql.plan.SparkWork) 
    at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) 
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656) 
    at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:238) 
    at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:226) 
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:745) 
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) 
    at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) 
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) 
    at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139) 
    at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) 
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) 
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) 
    at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) 
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) 
    at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:131) 
    at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) 
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) 
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) 
    at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) 
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) 
    at org.apache.hadoop.hive.ql.exec.spark.KryoSerializer.deserialize(KryoSerializer.java:49) 
    at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:318) 
    at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:366) 
    at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:335) 
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 

使用蜂巢我可以做簡單的選擇,但需要火花所有其他操作與Error: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask (state=08S01,code=3)控制檯和錯誤以上的紗線日誌結束。 現在我的每個蜂巢數據庫都癱瘓了(我很少)。我一整天都在試圖解決這個問題,但是不能做一些事情(蜂房重啓,紗節點重啓,更換紗主)。

您認爲導致問題的原因是什麼?如何解決?

回答

1

我想通了。

重新啓動hive-server2一小段時間後,而不是得到錯誤:org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 26我得到錯誤:org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.hadoop.hive.ql.io.RCFileOutputFormat。用第二種形式很明顯,在節點上執行的spark沒有在classpath上有一些jar。我不知道原因,爲什麼在一瞬間發生火花無法加載這些罐子,而是在將它們手動複製到每個節點上的lib文件夾並重新啓動節點後,一切都恢復正常。