apache-spark

6201 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
How to efficiently remove duplicate rows in Spark Dataframe, keeping row with highest timestamp
user_8738940
•
3
2
183
apache-spark scala sql
Load only struct from map's value from an avro file into a Spark Dataframe
user_71549140
•
1
1
93
pyspark apache-spark-sql apache-spark struct dictionary
How to create variable with the first day of current month in Databricks Spark Scala?
user_89164080
•
3
2
164
databricks apache-spark java-time scala java
Airflow doesn't pick up FAILED status of Spark job
user_31281560
•
3
0
158
airflow kubernetes-helm apache-spark kubernetes
How to create dataframe with struct column in PySpark without specifying a schema?
user_25403360
•
1
1
1096
pyspark-schema pyspark apache-spark-sql apache-spark struct
Spark 3.2 on Kubernetes keeps throwing okhttp3/okio EOFException
user_46491430
•
2
0
108
fabric8 okhttp apache-spark kubernetes
Azure Databricks cluster spark configuration is disabled
user_12320870
•
2
1
443
azure-databricks databricks apache-spark
Starting a pyspark session take a lot of time
user_167537660
•
2
2
340
pyspark apache-spark python
pandas udf as a window function in pyspark
user_49187650
•
1
1
209
pyspark pandas apache-spark sql python
How can I increase spark.driver.memoryOverhead in Google dataproc?
user_151733840
•
3
2
140
memory-overhead google-cloud-dataproc apache-spark executor
Hot Questions