2016-08-15 119 views
3

我使用tar球安裝了Cassandra-2.2.7。 Cassandra工作正常,我在一個keyspace中創建了表。現在我想將這個表格數據轉換成HDFS。我正在使用Sqoop-1.4.6。我一直在下面的庫文件在$ SQOOP_HOME/lib中/,嘗試使用Sqoop將數據從Cassandra傳輸到Hadoop時發生錯誤

apache-cassandra-2.2.7.jar 
apache-cassandra-thrift-2.2.7.jar 
cassandra-jdbc-1.2.5.jar 
cassandra-all-1.2.0.jar 
libthrift-0.8.0.jar 
thrift-server-0.3.7.jar 

然後我跑的命令,

bin/sqoop list-tables --driver org.apache.cassandra.cql.jdbc.CassandraDriver --connect jdbc:cassandra://127.0.0.1:7199/sample --username cassandra --password cassandra 

當我運行此命令時,我得到以下錯誤,

Warning: /usr/lib/sqoop/../hcatalog does not exist! HCatalog jobs will fail. 
 
Please set $HCAT_HOME to the root of your HCatalog installation. 
 
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail. 
 
Please set $ACCUMULO_HOME to the root of your Accumulo installation. 
 
16/08/15 16:52:41 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 
 
16/08/15 16:52:41 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 
 
16/08/15 16:52:41 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time. 
 
16/08/15 16:52:41 INFO manager.SqlManager: Using default fetchSize of 1000 
 
SLF4J: Class path contains multiple SLF4J bindings. 
 
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/hadoop-1.2.1/lib/sf2hadoop.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
 
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/hadoop-1.2.1/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
 
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 
 
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] 
 
16/08/15 16:52:41 ERROR manager.SqlManager: Error reading database metadata: java.sql.SQLNonTransientConnectionException: org.apache.thrift.transport.TTransportException 
 
java.sql.SQLNonTransientConnectionException: org.apache.thrift.transport.TTransportException 
 
\t at org.apache.cassandra.cql.jdbc.CassandraConnection.<init>(CassandraConnection.java:159) 
 
\t at org.apache.cassandra.cql.jdbc.CassandraDriver.connect(CassandraDriver.java:92) 
 
\t at java.sql.DriverManager.getConnection(DriverManager.java:664) 
 
\t at java.sql.DriverManager.getConnection(DriverManager.java:247) 
 
\t at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:885) 
 
\t at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52) 
 
\t at org.apache.sqoop.manager.SqlManager.listTables(SqlManager.java:520) 
 
\t at org.apache.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49) 
 
\t at org.apache.sqoop.Sqoop.run(Sqoop.java:143) 
 
\t at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) 
 
\t at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) 
 
\t at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) 
 
\t at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) 
 
\t at org.apache.sqoop.Sqoop.main(Sqoop.java:236) 
 
Caused by: org.apache.thrift.transport.TTransportException 
 
\t at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) 
 
\t at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) 
 
\t at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) 
 
\t at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) 
 
\t at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) 
 
\t at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) 
 
\t at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) 
 
\t at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) 
 
\t at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) 
 
\t at org.apache.cassandra.thrift.Cassandra$Client.recv_describe_cluster_name(Cassandra.java:1247) 
 
\t at org.apache.cassandra.thrift.Cassandra$Client.describe_cluster_name(Cassandra.java:1235) 
 
\t at org.apache.cassandra.cql.jdbc.CassandraConnection.<init>(CassandraConnection.java:130) 
 
\t ... 13 more 
 
Could not retrieve tables list from server 
 
16/08/15 16:52:41 ERROR tool.ListTablesTool: manager.listTables() returned null

我以爲這是某種節儉的rror,但無法糾正它,即使我用其他幾個版本替換了lib-thrift jar。

回答

2

Cassandra JDBC驅動程序非常舊。上次我嘗試使用它,我需要手動編譯它對付最新的cassandra thrift庫並修復一些編譯錯誤。

我會建議使用一個Spark工作來傳輸數據,或者如果你必須使用sqoop,這取決於你正在做的DataStax企業什麼包含自定義sqoop對Cassandra的驅動程序。