如何知道Spark應用程序的執行者IP？

Question

我想在運行時獲取執行程序的所有IP地址，我應該使用Spark中的哪個API？或者在運行時獲取IP的任何其他方法？

Answer 1

Apache Spark中有一個類，即ExecutorInfo，它具有返回執行主機IP的方法executorHost（）。

Answer 2

您應該使用SparkListener抽象類並攔截兩個執行程序特定事件 - SparkListenerExecutorAdded和SparkListenerExecutorRemoved。

override def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit = { 
    val execId = executorAdded.executorId 
    val host = executorAdded.executorInfo.executorHost 
    executors += (execId -> host) 
    println(s">>> executor id=$execId added on host=$host") 
} 

override def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit = { 
    val execId = executorRemoved.executorId 
    val host = executors remove execId getOrElse "Host unknown" 
    println(s">>> executor id=$execId removed from host=$host") 
} 

整個工作項目在我的Spark Executor Monitor Project。