apache-spark

6201 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
Optimizing "withColumn when otherwise" performance in pyspark
user_76292570
•
2
2
876
pyspark databricks apache-spark-sql apache-spark
Spark history server - find part of SQL query for stage
user_8363180
•
2
0
92
apache-spark-sql apache-spark
Looping the files inside the folder and load data into delta table using databricks
user_171556060
•
2
1
392
delta-lake databricks apache-spark
Scala - looping over case class names to use as type parameters
user_8363180
•
3
2
170
apache-spark scala
When was automatic Spark RDD partition cache eviction implemented?
user_15869650
•
2
1
49
apache-spark scala
How filter PySpark DataFrame with PySpark for current date
user_155206150
•
2
1
1255
pyspark apache-spark
How to load data at Spark Dataframe in each Worker to prevent loading huge data to Master node
user_66405040
•
2
1
591
apache-spark-sql apache-spark scala
Is there any stable method on the SparkSession/SparkContext/RDD that we can call to easily detect when eviction is happening?
user_15869650
•
1
1
103
apache-spark scala
Databricks Delta Lake - Reading data from JSON file
user_63406090
•
2
1
1023
delta-lake pyspark databricks apache-spark-sql apache-spark
AWS EMR: Pyspark: Rdd: mappartitions: Could not find valid SPARK_HOME while searching: Spark closures
user_73439220
•
7
2
520
amazon-emr python-requests pyspark apache-spark-sql apache-spark
Hot Questions