0
嘗試使用啓用Kerberos的Apache Kafka(0.9)與Apache spark 1.6.3時發生錯誤.Zookeeper版本爲3.4.5 我必須連接兩個kafka。一個是啓用keberos,另一個不是,所以我沒有在spark executor的額外java opts中設置java.security.auth.login.config屬性。Apache Kafka帶火花的安全和非安全連接1.6.3
Kafka Initialization failed: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:648)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:542)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:524)
at com.spark.receiver.helper.KafkaChannelHelper.initializeConnection(KafkaChannelHelper.java:277)
at com.spark.receiver.helper.KafkaChannelHelper$2.run(KafkaChannelHelper.java:240)
Caused by: org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in `/home/user/kafka_client.conf`.
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:74)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:60)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:79)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:577)
... 4 more
Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in `/home/user/kafka_client.conf`.
at org.apache.kafka.common.security.kerberos.Login.login(Login.java:294)
at org.apache.kafka.common.security.kerberos.Login.<init>(Login.java:104)
at org.apache.kafka.common.security.kerberos.LoginManager.<init>(LoginManager.java:44)
at org.apache.kafka.common.security.kerberos.LoginManager.acquireLoginManager(LoginManager.java:85)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:55)
... 7 more
java.security.auth.login.config被設置在消費者,其連接到kafkaConsumer itself.The代碼是:
public void initializeConnection() {
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT");
System.setProperty("java.security.auth.login.config", jassFilePath);
try {
this.consumer = new KafkaConsumer<String, byte[]>(props);
} catch (Exception e) {
LOGGER.error("Kafka Initialization failed: ", e);
}
}
kafka_client.conf只包含以下部分:
KafkaClient{
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
keyTab="/etc/security/keytabs/user.keytab"
storeKey=true
principal="[email protected]"
serviceName="kafka";
};