I'm developing a complex pipeline in Vertex AI using Pipelines and components. I would like to import some custom modules and functions I developed for this use case. Unfortunately, I cannot figure out how to import those custom functions in my code without creating ad-hoc Docker images or without publishing my code on public repositories like PyPi.
There are two pain points in pasting those custom functions' code in each component:
- The code becomes huge and difficult read
- The function's code completely loses the maintenability because at each small change, I have to replicate it for each component.