Changing resource file dynamically without repackaging jar in Scala

userE5Dm 0
Viewed 31

I have a Maven scala project with a config file located here: src/main/resources/reference.yaml. This file contains particular configurable parameters that are used throughout the entire application.

Currently, I package the jar and read the file in the following manner, which is found in the classpath.

import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import java.io.{File, InputStream}

val configInput: InputStream = getClass.getResourceAsStream("/" + configFileName)
val mapper = new ObjectMapper(new YAMLFactory)
mapper.registerModule(DefaultScalaModule)
mapper.readValue(configInput, classOf[Map[String, Any]])

I have custom functions to be able to parse the yaml file and load the variables into an object that stores all my config values.

object Paths {

  // Config
  val table: String = getConfigValue("sql.finance_table", "finance_v1")
}

This object is imported in various other scala files that define the pipeline, and use the config parameters to follow the specification of the user.

To execute the pipeline, the jar is packaged up and deployed to a cluster on databricks where it runs as part of a job. Thus, all config values are static and provided during build time.

While this works, I would like to separate the resource file customisation and the packaging and deployment of the jar. Is it possible to simply get the jar to read a resource file from an external location? This would allow the user to modify the config even after the jar has been built.

I'm not using lightbend or pureconfig.

config maven java databricks scala
1 Answers

There are several options:

  • add an "external" file in the classpath when running the code. The classpath does not only consists of the JARs of your app, it can also contain regular files. This way you don't even have to modify your current code. But.. I don't know if/how to customize the classpath with Databricks.

  • modify your code to read from "external" file in the first place rather than the classpath. You could hardcodd the path of the file or make it configurable via a system property.

Gaël J0
Related