在Spark Scala中將數據框轉換爲配置單元表

Question

我試圖將Spark中的數據框轉換爲配置單元表。我從一個XML文件讀取了一個數據框。它使用SQL上下文來執行此操作。我想將此數據框保存爲配置表。我收到此錯誤：

"WARN HiveContext$$anon$1: Could not persist database_1 . test_table in a Hive compatible way. Persisting it into Hive metastore in Spark SQL specific format."

object spark_conversion { 
def main(args: Array[String]): Unit = { 

if (args.length < 2) { 
    System.err.println("Usage: <input file> <output dir>") 
    System.exit(1) 
} 
val in_path = args(0) 
val out_path_csv = args(1) 
val conf = new SparkConf() 
     .setMaster("local[2]") 
     .setAppName("conversion") 
val sc = new SparkContext(conf) 

val hiveContext = new HiveContext(sc) 

val df = hiveContext.read 
    .format("com.databricks.spark.xml") 
    .option("rowTag", "PolicyPeriod") 
    .option("attributePrefix", "attr_") 
    .load(in_path) 

df.write 
    .format("com.databricks.spark.csv") 
    .option("header", "true") 
    .save(out_path_csv) 

df.saveAsTable("database_1.test_table") 

df.printSchema() 
df.show() 

Answer 1

saveAsTable火花不與蜂巢兼容。我在CDH 5.5.2上。從Cloudera的網站解決方案：

df.registerTempTable(tempName) 
hsc.sql(s""" 
CREATE TABLE $tableName (  
// field definitions ) 
STORED AS $format """) 
hsc.sql(s"INSERT INTO TABLE $tableName SELECT * FROM $tempName") 

http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_spark_ki.html