How can I configure spark-submit (or DataProc) to download maven dependencies (jars) from GitHub packages?

user_38368 0
Viewed 965

I am trying to get spark-submit (via GCP DataProc) to download maven dependencies from a GitHub packages repository.

Adding spark.jars.repositories=https://myuser:mytoken@maven.pkg.github.com/myorg/my-maven-packages-repo/ to the spark-submit command doesn't help...

The correct urls are accessed, but the files aren't downloaded (https://maven.pkg.github.com/myorg/my-maven-packages.repo/myorg/mylibrary/1.0.0/library-1.0.0.jar).

How can I get this working? (Without using uber-jars!)

spark-submit github-package-registry google-cloud-dataproc apache-spark ivy
1 Answers

Instead of adding:

spark.jars.repositories=https://myuser:mytoken@maven.pkg.github.com/myorg/my-maven-packages-repo/ to the spark-submit command

add:

--files=gs://my-bucket/github-ivysettings.xml

and

spark.jars.ivySettings=github-ivysettings.xml

upload the following file (github-ivysettings.xml) to the bucket:

<ivysettings>  

  <settings defaultResolver="default"/>

  <include url="${ivy.default.settings.dir}/ivysettings-public.xml"/>
  <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
  <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
  <include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
  <include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>

  <credentials
      host="maven.pkg.github.com" realm="GitHub Package Registry"
      username="myuser" passwd="mytoken"
      />
  <resolvers>
      <ibiblio
          name="private-github"
          m2compatible="true" useMavenMetadata="true" usepoms="true"
          root="https://maven.pkg.github.com/myorg/my-maven-packages-repo/"
          pattern="[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"
      />
    <chain name="default" returnFirst="true" checkmodified="true">
      <resolver ref="local" />
      <resolver ref="shared" />
      <resolver ref="public" />
      <resolver ref="private-github" />
    </chain>
  </resolvers>
</ivysettings>

This will preserve the current search order (local, shared, public) and then search in your private repository.

Note that the realm is important, so if you use this for a different private repository, change the host, the root and the realm.

user_383680
Related