修改火花數據幀列

Question

我有一個火花數據幀，我想添加一個具有特定值的新列。我曾嘗試使用withcolumn函數，但它不按預期工作。我想與特定值的新列或我要替換現有列

Answer 1

見這個例子

我有一個數據幀：

>>> df.show() 
+-------+----+-----+---+ 
| name|year|month|day| 
+-------+----+-----+---+ 
| Ali|2014| 9| 1| 
| Matei|2015| 10| 26| 
|Michael|2015| 10| 25| 
|Reynold|2015| 10| 25| 
|Patrick|2015| 9| 1| 
+-------+----+-----+---+ 

我想補充一個信息，對於每一行，我可以使用​​3210做

>>> from pyspark.sql.functions import lit 
>>> df.withColumn('my_new_column', lit('testing info for all')).show() 
+-------+----+-----+---+--------------------+ 
| name|year|month|day|  my_new_column| 
+-------+----+-----+---+--------------------+ 
| Ali|2014| 9| 1|testing info for all| 
| Matei|2015| 10| 26|testing info for all| 
|Michael|2015| 10| 25|testing info for all| 
|Reynold|2015| 10| 25|testing info for all| 
|Patrick|2015| 9| 1|testing info for all| 
+-------+----+-----+---+--------------------+ 

如果你想添加不同的信息，每行一個列表，你可以使用explode：

>>> from pyspark.sql.functions import explode 
>>> df.withColumn('my_new_column', 
...    explode(array(lit('testing info for all'), 
...        lit('other testing again')))).show() 
+-------+----+-----+---+--------------------+ 
| name|year|month|day|  my_new_column| 
+-------+----+-----+---+--------------------+ 
| Ali|2014| 9| 1|testing info for all| 
| Ali|2014| 9| 1| other testing again| 
| Matei|2015| 10| 26|testing info for all| 
| Matei|2015| 10| 26| other testing again| 
|Michael|2015| 10| 25|testing info for all| 
|Michael|2015| 10| 25| other testing again| 
|Reynold|2015| 10| 25|testing info for all| 
|Reynold|2015| 10| 25| other testing again| 
|Patrick|2015| 9| 1|testing info for all| 
|Patrick|2015| 9| 1| other testing again| 
+-------+----+-----+---+--------------------+