如何查找pyspark數據幀內存使用情況？

Question

對於python數據框，info（）函數提供內存使用情況。 pyspark有什麼等價物嗎？ 感謝

Answer 1

嘗試使用the following trick：

import py4j.protocol 
from py4j.protocol import Py4JJavaError 
from py4j.java_gateway import JavaObject 
from py4j.java_collections import JavaArray, JavaList 

from pyspark import RDD, SparkContext 
from pyspark.serializers import PickleSerializer, AutoBatchedSerializer 

# your dataframe what you'd estimate 
df 

# Helper function to convert python object to Java objects 
def _to_java_object_rdd(rdd): 
    """ Return a JavaRDD of Object by unpickling 
    It will convert each Python object into Java object by Pyrolite, whenever the 
    RDD is serialized in batch or not. 
    """ 
    rdd = rdd._reserialize(AutoBatchedSerializer(PickleSerializer())) 
    return rdd.ctx._jvm.org.apache.spark.mllib.api.python.SerDe.pythonToJava(rdd._jrdd, True) 

# First you have to convert it to an RDD 
JavaObj = _to_java_object_rdd(df.rdd) 

# Now we can run the estimator 
sc._jvm.org.apache.spark.util.SizeEstimator.estimate(JavaObj)