apache-spark

6201 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
How do I read a text file & apply a schema with PySpark?
user_13547120
•
4
1
7921
pyspark apache-spark python
Dask dataframe concatenating parquet files throws out of memory
user_115905820
•
2
1
1155
dask-dataframe dask pyarrow apache-spark python
What happens when you restart a spark job if it encounters unexpected format in the data fed to kafka
user_134977920
•
3
1
469
spark-structured-streaming apache-kafka apache-spark
Joining two clustered tables in spark dataset seems to end up with full shuffle
user_8507370
•
2
1
252
apache-spark-dataset apache-spark scala
How to use pivot in SQL (not as DataFrame grouping operator)?
user_31900180
•
4
1
553
apache-spark-sql apache-spark scala sql
How can I deploy HDFS (Hadoop Distributed FS) to a K8s (Kubernetes) cluster?
user_81358610
•
3
1
3646
hdfs apache-spark kubernetes hadoop
NotImplementedError: Invalid returnType with grouped map Pandas UDFs
user_59279030
•
4
1
1224
pyspark pandas apache-spark python
Switching between Databricks Connect and local Spark environment
user_10852910
•
8
0
604
databricks-connect pyspark databricks apache-spark
Dataproc on GKE doesn't use image when submitting the job
user_124258550
•
2
1
230
google-kubernetes-engine google-cloud-dataproc apache-spark google-cloud-platform
How to read a compressed Spark eventLog?
user_36222830
•
3
2
1018
lz4 apache-spark scala
Hot Questions