Impala +實木複合地板文件

Question

我創建了實木複合地板文件，然後嘗試將其導入Impala表。

我創建的表如下：

CREATE EXTERNAL TABLE `user_daily` (
`user_id` BIGINT COMMENT 'User ID', 
`master_id` BIGINT, 
`walletAgency` BOOLEAN, 
`zone_id` BIGINT COMMENT 'Zone ID', 
`day` STRING COMMENT 'The stats are aggregated for single days', 
`clicks` BIGINT COMMENT 'The number of clicks', 
`impressions` BIGINT COMMENT 'The number of impressions', 
`avg_position` BIGINT COMMENT 'The average position * 100', 
`money` BIGINT COMMENT 'The cost of the clicks, in hellers', 
`web_id` BIGINT COMMENT 'Web ID', 
`discarded_clicks` BIGINT COMMENT 'Number of discarded clicks from column "clicks"', 
`impression_money` BIGINT COMMENT 'The cost of the impressions, in hellers' 
) 
PARTITIONED BY (
year BIGINT, 
month BIGINT 
) 
STORED AS PARQUET 
LOCATION '/warehouse/impala/contextstat.db/user_daily/'; 

然後我有複製文件與此架構：

parquet-tools schema user_daily/year\=2016/month\=8/part-r-00001-fd77e1cd-c824-4ebd-9328-0aca5a168d11.snappy.parquet 
message spark_schema { 
    optional int32 user_id; 
    optional int32 web_id (INT_16); 
    optional int32 zone_id; 
    required int32 master_id; 
    required boolean walletagency; 
    optional int64 impressions; 
    optional int64 clicks; 
    optional int64 money; 
    optional int64 avg_position; 
    optional double impression_money; 
    required binary day (UTF8); 
} 

，然後當我嘗試看到

SELECT * FROM user_daily; 

我的條目獲得

File 'hdfs://.../warehouse/impala/contextstat.db/user_daily/year=2016/month=8/part-r-00000-fd77e1cd-c824-4ebd-9328-0aca5a168d11.snappy.parquet' 
has an incompatible Parquet schema for column 'contextstat.user_daily.user_id'. 
Column type: BIGINT, Parquet schema: 
optional int32 user_id [i:0 d:1 r:0] 

你知道如何解決這個問題嗎？我認爲BIGINT和int_32是一樣的。我應該改變桌子的方案還是改變鑲木地板的文件？

Answer 1

我用CAST(... AS BIGINT)，這改變拼花模式從int32到int64。然後我必須重新排序列，因爲它不會按名稱加入。然後它工作。

Answer 2

BIGINT是int64，這就是爲什麼它抱怨。但是，您不一定必須弄清楚自己必須使用的不同類型，因此Impala可以爲您做到這一點。只需使用CREATE TABLE LIKE PARQUET變種：

The variation CREATE TABLE ... LIKE PARQUET 'hdfs_path_of_parquet_file' lets you skip the column definitions of the CREATE TABLE statement. The column names and data types are automatically configured based on the organization of the specified Parquet data file, which must already reside in HDFS.