pyspark

4047 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
Multiple executors on same worker fail in Spark standalone cluster (python scripts)
user_80942950
•
2
0
265
pyspark apache-spark
simple spark-submit job fails on EMR 5.5.0 with FileNotFoundException for __spark_config__.zip on HDFS
user_74916200
•
2
1
504
emr hadoop-yarn pyspark apache-spark python
How to write pyspark dataframe to HDFS and then how to read it back into dataframe?
user_57648830
•
15
1
28306
spark-dataframe hdfs pyspark hadoop python
ERROR:SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063
user_63769350
•
3
3
9719
pyasn1 pyspark python-2.7 lambda python
Apply sklearn trained model on a dataframe with PySpark
user_57065480
•
8
3
5503
scikit-learn pyspark apache-spark python
How to read only n rows of large CSV file on HDFS using spark-csv package?
user_36705320
•
20
7
32790
spark-csv hdfs pyspark apache-spark-sql apache-spark
How to change column metadata in PySpark?
user_8678890
•
6
2
7979
apache-spark-ml pyspark apache-spark dataframe metadata
How can I use a function in dataframe withColumn function in Pyspark?
user_47820600
•
2
1
12058
pyspark apache-spark dataframe function replace
Custom Evaluator during cross validation SPARK
user_56724850
•
3
1
1245
cross-validation pyspark
How to read specific column in pyspark?
user_46848610
•
3
2
8908
pyspark pandas python
Hot Questions