如何訪問以編程方式啓用Kerberos的hadoop羣集？

Question

我有這樣一段代碼可以從一個Hadoop文件系統獲取的文件。我在單個節點上設置hadoop，並從本地機器運行此代碼，以查看它是否能夠從該節點上的HDFS設置中獲取文件。有效。

package com.hdfs.test.hdfs_util; 
/* Copy file from hdfs to local disk without hadoop installation 
* 
* params are something like 
* hdfs://node01.sindice.net:8020 /user/bob/file.zip file.zip 
* 
*/ 

import java.io.IOException; 

import org.apache.hadoop.conf.Configuration; 

import org.apache.hadoop.fs.FileSystem; 

import org.apache.hadoop.fs.Path; 

public class HDFSdownloader{ 

public static void main(String[] args) throws Exception { 

    System.getProperty("java.classpath"); 
    if (args.length != 3) { 

     System.out.println("use: HDFSdownloader hdfs src dst"); 

     System.exit(1); 

    } 

    System.out.println(HDFSdownloader.class.getName()); 
    HDFSdownloader dw = new HDFSdownloader(); 

    dw.copy2local(args[0], args[1], args[2]); 

} 

private void copy2local(String hdfs, String src, String dst) throws IOException { 

    System.out.println("!! Entering function !!"); 
    Configuration conf = new Configuration(); 
    conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()); 
    conf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName()); 


    conf.set("fs.default.name", hdfs); 

    FileSystem.get(conf).copyToLocalFile(new Path(src), new Path(dst)); 

    System.out.println("!! copytoLocalFile Reached!!"); 

} 

} 

現在我把相同的代碼捆綁在一個jar中，並試圖在另一個節點（比如說B）上運行它。這次代碼必須從適當的分佈式Hadoop集羣中獲取文件。該羣集中啓用了Kerberos。

的代碼運行，但給了一個例外：

Exception in thread "main" org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] 
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) 
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) 
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) 
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) 
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2115) 
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) 
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) 
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) 
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317) 
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) 
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289) 
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2030) 
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1999) 
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1975) 
at com.hdfs.test.hdfs_util.HDFSdownloader.copy2local(HDFSdownloader.java:49) 
at com.hdfs.test.hdfs_util.HDFSdownloader.main(HDFSdownloader.java:35) 

有沒有辦法來編程使此代碼運行。出於某種原因，我無法在源節點上安裝kinit。

Answer 1

這裏的代碼片段在你上面，即所描述編程訪問啓用了Kerberos的集羣方案工作。要點需要注意的是

• 在UserGroupInformation提供密鑰表文件位置
• 在JVM參數提供Kerberos領域的細節 - krb5.conf文件

• 定義Hadoop的安全認證方式爲Kerberos的

  import org.apache.hadoop.conf.Configuration; 
  import org.apache.hadoop.fs.FileStatus; 
  import org.apache.hadoop.fs.FileSystem; 
  import org.apache.hadoop.fs.Path; 
  import org.apache.hadoop.security.UserGroupInformation; 
  
  public class KerberosHDFSIO { 
  
  public static void main(String[] args) throws IOException { 
  
      Configuration conf = new Configuration(); 
      //The following property is enough for a non-kerberized setup 
      //  conf.set("fs.defaultFS", "localhost:9000"); 
  
      //need following set of properties to access a kerberized cluster 
      conf.set("fs.defaultFS", "hdfs://devha:8020"); 
      conf.set("hadoop.security.authentication", "kerberos"); 
  
      //The location of krb5.conf file needs to be provided in the VM arguments for the JVM 
      //-Djava.security.krb5.conf=/Users/user/Desktop/utils/cluster/dev/krb5.conf 
  
      UserGroupInformation.setConfiguration(conf); 
      UserGroupInformation.loginUserFromKeytab("user@HADOOP_DEV.ABC.COM", 
        "/Users/user/Desktop/utils/cluster/dev/.user.keytab"); 
  
      try (FileSystem fs = FileSystem.get(conf);) { 
       FileStatus[] fileStatuses = fs.listStatus(new Path("/user/username/dropoff")); 
       for (FileStatus fileStatus : fileStatuses) { 
        System.out.println(fileStatus.getPath().getName()); 
       } 
      } catch (IOException e) { 
       // TODO Auto-generated catch block 
       e.printStackTrace(); 
      } 
  } 
  

  }