I know that Kubeflow only modifies the container with the specified libraries to be installed. But I want to use my custom module in the training Component section of the pipeline.
So let me clarify my case; I'm deploying a GCP Vertex AI pipeline which exists of preprocessing and training steps. And there is also custom library that I created using some libraries like scikit. My main issue is that I want to re-use that library objects within my training step which looks like;
packages_to_install = [
"pandas",
"sklearn",
"mycustomlibrary?"
],
)
def train_xgb_model(
dataset: Input[Dataset],
model_artifact: Output[Model]
):
from MyCustomLibrary import XGBClassifier
import pandas as pd
data = pd.read_csv(dataset.path)
model = XGBClassifier(
objective="binary:logistic"
)
model.fit(
data.drop(columns=["target"]),
data.target,
)
score = model.score(
data.drop(columns=["target"]),
data.target,
)
model_artifact.metadata["train_score"] = float(score)
model_artifact.metadata["framework"] = "XGBoost"
model.save_model(model_artifact.path)```