Unable to pass java library path with beam multi-language pipelines

user_9648514 0
Viewed 251

We are using a beam multi-language pipeline using python and java(ref https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/). We are creating a cross-language pipeline using java. We have some external jar files that required a java library path. Code gets compiled properly and is able to create a jar file. When I run the jar file it creates a Grpc server but when I use the python pipeline to call External transform it is not picking up the java library path it picks the default java library path. jni_emdq required library path to overwrite

Tried -Djava.library.path=<path_to_dll> while running jar file. Tried System.setProperty(“java.library.path”, “/path/to/library”). (Ref https://examples.javacodegeeks.com/java-library-path-what-is-java-library-and-how-to-use/)

Tried JvmInitializer of beam to overwrite system property. (Ref https://examples.javacodegeeks.com/java-library-path-what-is-java-library-and-how-to-use/)

Tried to pull code beam open source and tried to overwrite system proprty before expansion starts. It overwrite but it is not picking correct java path when calls using python external transform. (ref https://github.com/apache/beam/blob/master/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java)

apache-beam grpc-java java
2 Answers

You may want to use Artifacts staging.

By default, Portable Runner uses a pipeline class path to find the artifacts for staging to Job Server. It’s also possible to stage only required artifacts with a pipeline option: --filesToStage=<comma separated jars list>

It takes more time to stage artifacts when a pipeline runs for the first time. For the next attempts of pipeline run, it will check if the artifacts for staging has not been changed (using hashing) since last time and it will use already staged(cached) artifacts if no other artifacts staging is required.

Also, it’s possible to avoid an artifact staging at all if all required jars and other files will be available in the class path on workers in case of using Process SDK Harness or in the class path in Docker image in case of using Docker SDK Harness. In this case we need to leave --filesToStage option empty.

user_106873250

It sounds like you're trying to stage native code, and not just Jar files. You could try using custom containers instead. You would have to make sure the expansion service that you start up is configured to use these containers (e.g. via pipeline options).

user_5823330
Related