如何檢查實例是否位於pyspark中的數據框中並從數據框中獲取出現？

Question

我有一個實例從具有3個不同屬性的數據幀中提取：Atr1，Atr2和Atr3。另一方面，我有一個包含4個屬性的數據框：Atr1，Atr2，Atr3，Atr4，但Atributes Atr1，Atr2和Atr3與前面提到的實例相同。我有這樣的事情：

Instance: 

[Row(Atr1=u'A', Atr2=u'B', Atr3=24)] 

Dataframe: 

+------+------+------+------+ 
| Atr1 | Atr2 | Atr3 | Atr4 | 
+------+------+------+------+ 
| 'C' | 'B' | 21 | 'H' | 
+------+------+------+------+ 
| 'D' | 'B' | 21 | 'J' | 
+------+------+------+------+ 
| 'E' | 'B' | 21 | 'K' | 
+------+------+------+------+ 
| 'A' | 'B' | 24 | 'I' | 
+------+------+------+------+ 

所以，有上述情況，我要檢查它是否在與屬性ATR1，ATR2和ATR3這些值數據幀存在一個實例，如果它存在，請Atr4的值。在這種情況下，'我'。

Answer 1

這是一個可接受的答案？

df[(df['Atr1'] == row.Atr1) & (df['Atr2'] == row.Atr2) & (df['Atr3'] == row.Atr3)].Atr4 

與row行和df你mentionned數據幀。

Answer 2

希望這有助於！

from pyspark.sql.types import Row 
from pyspark.sql.functions import col 

#sample data 
row_list = [Row(Atr1=u'A', Atr2=u'B', Atr3=24), 
      Row(Atr1=u'E', Atr2=u'F', Atr3=20),] 
df = sc.parallelize([('C', 'B', 21, 'H'), 
        ('D', 'B', 21, 'J'), 
        ('E', 'B', 21, 'K'), 
        ('A', 'B', 24, 'I')]).\ 
    toDF(["Atr1", "Atr2", "Atr3", "Atr4"]) 

search_df = df.join(sqlContext.createDataFrame(row_list), ["Atr1", "Atr2", "Atr3"], "right").\ 
    withColumn("rowItem_Exist", col('Atr4').isNotNull()) 
search_df.show() 

輸出是：

+----+----+----+----+-------------+ 
|Atr1|Atr2|Atr3|Atr4|rowItem_Exist| 
+----+----+----+----+-------------+ 
| E| F| 20|null|  false| 
| A| B| 24| I|   true| 
+----+----+----+----+-------------+