Python OOP in Kubeflow Pipelines

user_18892438 0
Viewed 82

I have a complete preprocessing, training, evaluating ML pipeline in Python 3.8 that I want to transform in a Kubeflow pipeline. It's mainly for segmentation task (i.e. on RGB images)

All my code is design in a OOP way. Is this possible to keep the OOP strategy in Kubeflow ? By this question I mean :

  • Passing objects through pipeline containers
  • Use the func_to_conntainer_op decorator to transform class methods to containers

If it's possible, if you have any code samples as exemples it will be helpful.

Otherwise, what do you recommend to move to KF easily ?
Thanks for your help ;)

kubeflow-pipelines kubeflow python
1 Answers

I don't know if there is a native Kubeflow way of doing this, but as func_to_container_op takes a standalone function as its input, the self state variable will surely be a problem. You might have some luck by inspecting the source code of the func_to_container_op function, which eventually uses a factory to create a class which you might create directly.

As a simpler alternative, if you look at the class-based API of ZenML, you can do something like this:


class Preprocessing(BaseStep):
    def entrypoint(self,
                   dataset: pd.DataFrame,
                   ) -> Output(train=pd.DataFrame, 
                               test=pd.DataFrame, 
                               validation=pd.DataFrame):
        
        ...
        return train_dataset, test_dataset, val_dataset

This step can then be put a in a Pipeline and deployed with Kubeflow as the orchestrator [see here]

As the step is already a class, perhaps there is an easy way for you to convert your existing classes to ZenML BaseStep classes?

Disclaimer: I'm one of the co-founders of ZenML.

user_15612320
Related