Spark, wasb and Jetty 11

user_9388056 0
Viewed 260

When running Pyspark 3.1.1 with zulu11.45.27-ca-jdk11.0.10-win_x64 and trying to access Azure wasbs://container@store.blob.core.windows.net using hadoop-azure-3.2.2.jar jetty-util-11.0.1.jar jetty-util-ajax-11.0.1.jar

I get the following error:

py4j.protocol.Py4JJavaError: An error occurred while calling o49.csv.
: java.lang.NoSuchMethodError: 'java.util.Properties org.eclipse.jetty.util.log.Log.getProperties()'
        at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createPermissionJsonSerializer(AzureNativeFileSystemStore.java:410)
        at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.<clinit>(AzureNativeFileSystemStore.java:314)
        at org.apache.hadoop.fs.azure.NativeAzureFileSystem.createDefaultStore(NativeAzureFileSystem.java:1443)
        at org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1368)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)

Any help would be appreciated. Thanks

pyspark azure hadoop jetty
1 Answers

Apparently the org.eclipse.jetty.util.log.Log class was deprecated on Jetty 10+.

I was able to solve this issue by using the latest version from 9.4.45, like:

jetty-util-9.4.45.v20220203.jar
jetty-util-ajax-9.4.45.v20220203.jar
jetty-server-9.4.45.v20220203.jar

I hope this works for you.

user_37208140
Related