使用Sqoop將數據從mysql導入HDFS

Question

我正在使用Hadoop-1.2.1和Sqoop-1.4.6。我使用sqoop使用該命令從數據庫meshtree導入表test到HDFS：

`sqoop import --connect jdbc:mysql://localhost/meshtree --username user --password password --table test` 

但是，它顯示了這個錯誤：

17/06/17 18:15:21 WARN tool.BaseSqoopTool: Setting your password on the  command-line is insecure. Consider using -P instead. 
17/06/17 18:15:21 INFO manager.MySQLManager: Preparing to use a MySQL  streaming resultset. 
17/06/17 18:15:21 INFO tool.CodeGenTool: Beginning code generation 
17/06/17 18:15:22 INFO manager.SqlManager: Executing SQL statement: SELECT  t.* FROM `test` AS t LIMIT 1 
17/06/17 18:15:22 INFO orm.CompilationManager: HADOOP_HOME is /home/student /Installations/hadoop-1.2.1/libexec/.. 
Note: /tmp/sqoop-student/compile/6bab6efaa3dc60e67a50885b26c1d14b/test.java  uses or overrides a deprecated API. 
Note: Recompile with -Xlint:deprecation for details. 
17/06/17 18:15:24 ERROR orm.CompilationManager: Could not rename /tmp/sqoop- student/compile/6bab6efaa3dc60e67a50885b26c1d14b/test.java to /home/student /Installations/hadoop-1.2.1/./test.java 
org.apache.commons.io.FileExistsException: Destination '/home/student /Installations/hadoop-1.2.1/./test.java' already exists 
at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:2378) 
at  org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:227) 
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:83) 
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:367) 
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:453) 
at org.apache.sqoop.Sqoop.run(Sqoop.java:145) 
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) 
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) 
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) 
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) 
at org.apache.sqoop.Sqoop.main(Sqoop.java:238) 
at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57) 
17/06/17 18:15:24 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop- student/compile/6bab6efaa3dc60e67a50885b26c1d14b/test.jar 
17/06/17 18:15:24 WARN manager.MySQLManager: It looks like you are importing  from mysql. 
17/06/17 18:15:24 WARN manager.MySQLManager: This transfer can be faster! Use  the --direct 
17/06/17 18:15:24 WARN manager.MySQLManager: option to exercise a MySQL- specific fast path. 
17/06/17 18:15:24 INFO manager.MySQLManager: Setting zero DATETIME behavior  to convertToNull (mysql) 
17/06/17 18:15:24 INFO mapreduce.ImportJobBase: Beginning import of test 
17/06/17 18:15:27 INFO mapred.JobClient: Cleaning up the staging area  hdfs://localhost:9000/home/student/Installations/hadoop-1.2.1/data/mapred /staging/student/.staging/job_201706171814_0001 
17/06/17 18:15:27 ERROR security.UserGroupInformation:  PriviledgedActionException as:student  cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory  test already exists 
17/06/17 18:15:27 ERROR tool.ImportTool: Encountered IOException running  import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output  directory test already exists 
at  org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileO utputFormat.java:137) 
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:973) 
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at  org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) 
at  org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936) 
at org.apache.hadoop.mapreduce.Job.submit(Job.java:550) 
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580) 
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141) 
at  org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:201) 
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:413) 
at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:97) 
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:380) 
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:453) 
at org.apache.sqoop.Sqoop.run(Sqoop.java:145) 
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) 
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) 
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) 
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) 
at org.apache.sqoop.Sqoop.main(Sqoop.java:238) 
at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57) 

有什麼辦法弄清楚這個問題？

Answer 1

• 您正在導入，但未提供hdfs的目標目錄。當我們沒有提供任何目標目錄時，sqoop只運行一次導入，並且用你的mysql表名在hdfs中創建目錄。

So your query

sqoop import --connect jdbc:mysql://localhost/meshtree --username user --password password --table test

this create a directory with the name test1 in hdfs

• 只需添加下面的腳本

sqoop import --connect jdbc:mysql://localhost/meshtree --username user --password password --table test --target-dir test1

希望充分的做工精細，只是指sqoop import and all related sqoop

Answer 2

如果您打算將Sqoop用於分佈式Hadoop羣集，則不要使用URL localhost，這一點很重要。您提供的連接字符串將用於整個MapReduce集羣中的TaskTracker節點;如果您指定文字名稱localhost，則每個節點將連接到不同的數據庫（或更可能，根本沒有數據庫）。 而應使用數據庫主機的完整主機名或IP地址，該主機名或IP地址可由所有遠程節點看到。

請訪問Sqoop文檔Connecting to a Database Server部分了解更多信息。

Answer 3

您沒有權限。所以請聯繫myql dba以授予您相同的權利。 或者你可以自己做，如果你有管理員訪問MySQL。

grant all privileges on databasename.* to 'username'@'%' identified by 'password'; 

* - 對於所有表 ％ - 從任何主機

上述語法是授予權限，用戶在MySQL server.In你的情況讓這將是： -

grant all privileges on meshtree.test to 'root'@'localhost' identified by 'yourpassword';