我正嘗試使用Pig讀取Hive表中的數據。詳細信息如下:閱讀Pig中的非字符串分區Hive表
- 蜂巢版本1.1
- 豬0.12
- 的Hadoop 2.6.0
- Cloudera的分銷5.4.4
蜂巢表模式:
map <string, string>
yyyy int
mm int
dd int
Partitions are yyyy(int), mm(int), dd(int)
豬代碼:
012用於運行input_data = LOAD ‘dbname.tablename'
USING org.apache.hive.hcatalog.pig.HCatLoader()
;
input_data_f = FILTER input_data BY yyyy == 2016 AND
mm == 7 AND
dd == 19
;
rmf input_data_dump;
STORE input_data_f INTO ‘input_data_dump';
命令:pig -useHCatalog -f ./read_input.pig
我碰到下面的錯誤。
Error:
Pig Stack Trace
---------------
ERROR 2017: Internal error creating job configuration.
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2017: Internal error creating job configuration.
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:873)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:298)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:190)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1334)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1319)
at org.apache.pig.PigServer.execute(PigServer.java:1309)
at org.apache.pig.PigServer.executeBatch(PigServer.java:387)
at org.apache.pig.PigServer.executeBatch(PigServer.java:365)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:478)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: MetaException(message:Filtering is supported only on partition keys of type string)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:61)
at org.apache.hive.hcatalog.pig.HCatLoader.setLocation(HCatLoader.java:125)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:498)
... 19 more
Caused by: MetaException(message:Filtering is supported only on partition keys of type string)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_filter_result$get_partitions_by_filter_resultStandardScheme.read(ThriftHiveMetastore.java)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_filter_result$get_partitions_by_filter_resultStandardScheme.read(ThriftHiveMetastore.java)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_filter_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions_by_filter(ThriftHiveMetastore.java:2132)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions_by_filter(ThriftHiveMetastore.java:2116)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsByFilter(HiveMetaStoreClient.java:1047)
at org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:113)
at org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
... 22 more
尋找在網絡上拉去https://issues.apache.org/jira/browse/HIVE-7164
在蜂房site.xml中設置hive.metastore.integral.jdo.pushdown
爲true,唯一的解決辦法?這是一個公司設置,所以我不確定是否可以對hive-site.xml進行更改,如果我讓管理員進行更改,會有任何副作用?
嘗試了以下內容:
嘗試1
set hive.metastore.integral.jdo.pushdown true;
input_data = LOAD ‘dbname.tablename'
USING org.apache.hive.hcatalog.pig.HCatLoader()
;
input_data_f = FILTER input_data BY yyyy == 2016 AND
mm == 7 AND
dd == 19
;
STORE input_data_f INTO ‘input_data_dump';
我看到這在日誌中:
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, DuplicateForEachColumnRewrite, GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier, PartitionFilterOptimizer]}
嘗試2
set hive.metastore.integral.jdo.pushdown true;
set pig.exec.useOldPartitionFilterOptimizer true;
input_data = LOAD ‘dbname.tablename'
USING org.apache.hive.hcatalog.pig.HCatLoader()
;
input_data_f = FILTER input_data BY yyyy == 2016;
input_data_f1 = FILTER input_data_f BY mm == 7;
input_data_f2 = FILTER input_data_f1 BY dd == 19;
STORE input_data_f2 INTO ‘input_data_dump';
我看到這在日誌中:
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, DuplicateForEachColumnRewrite, GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier, NewPartitionFilterOptimizer]}
嘗試3
set pig.exec.useOldPartitionFilterOptimizer true;
input_data = LOAD ‘dbname.tablename'
USING org.apache.hive.hcatalog.pig.HCatLoader()
;
input_data_f = FILTER input_data BY yyyy == 2016;
input_data_f1 = FILTER input_data_f BY mm == 7;
input_data_f2 = FILTER input_data_f1 BY dd == 19;
STORE input_data_f2 INTO ‘input_data_dump';
我看到這在日誌中:
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, DuplicateForEachColumnRewrite, GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier, NewPartitionFilterOptimizer]}
通過上述的嘗試我仍然得到同樣的錯誤。
欣賞幫助。
感謝您的快速響應。我試着在腳本中設置屬性,但我仍然得到「僅在類型字符串的分區鍵上支持篩選」錯誤。我也嘗試在.pigbootup文件中設置它,但同樣的錯誤再次。 – Arnkrishn
在Pig 0.12.0中,Pig僅將第一個過濾器推送到裝載程序。你會得到相同的結果,但由於它會導致性能下降。 - 爲了解決這個問題,你應該爲所有分區使用一個過濾器語句。或者你可以指定: 'pig.exec.useOldPartitionFilterOptimizer = true' [見這裏的deails - 已知問題0.12](https://cwiki.apache.org/confluence/display/PIG/Pig+0.12.0+Known+問題) - 也在回答中更新了此評論。 –
謝謝!我試着將過濾器語句拆分成3個過濾器語句,並在腳本中將pig.exec.useOldPartitionFilterOptimizer標誌設置爲true,但仍然是相同的錯誤。我有,「set hive.metastore.integral.jdo.pushdown true;」 「set pig.exec.useOldPartitionFilterOptimizer true;」在腳本中。 – Arnkrishn