Setup of Flask architecture for machine learning pipeline

Viewed 108

I want to setup a machine learning pipeline that is callable by flask but I am facing some issues, these links are for the documentations I have read so far:

Let me explain the pipeline I have in mind:

  1. pull a dataframe from a PostgreSQL database

  2. encode said dataframe to make it ready for most algorithms

  3. split up the data
  4. feed to a pipeline and determine accuracy
  5. store the model in a pickle file

What is working so far:

All parts are working as a regular script I can just slap all the steps into one huge flask file with one decorator and it would run as well (my emergency solution)

The File Structure

The encoder script:

#Flask main thread
#makes flask start this part as application and not as module
app = Flask('encoder_module')

@app.route('/df_encoder')

def df_encoder(rng = 4):
     encoding stuff
    `return df`

The Pipeline script (random forest regressor here)

app = Flask('pipeline_module')

@app.route('/pipeline_rfr')

def pipeline_rfr():
     pipeline stuff
    `return grid_search_rfr`

The pickle module:

app = Flask('pickle_module')

@app.route('/store_reg_pickle')

def store_pickle():


    """
    Usage of a Pickle Model -Storage of a trained Model
    """
    model = grid_search_rfr
    #specify file name in letter strings
    model_file = "regression_model"
    with open(model_file, mode='wb') as m_f:
        pickle.dump(model, m_f)
    print(f"Model saved in: {os.getcwd()}")
    return model_file

The Main Flask File

#packages
from flask import Flask
from encoder_main_thread import df_encoder
from rfr_pipeline_function import pipeline_rfr
from pickle_call import store_pickle

app = Flask(__name__.split('.')[0])

@app.route('/regression_pipe')
@df_encoder
@pipeline_rfr
@store_reg_pickle
def regression_pipe():
   `return 'pipeline done`

The problem iss that the return value of the encoder cannot be a dataframe, only a string, tuple, etc.

Is there a workaround for this?

I actually want it to be a flawless passing of the dataframe to the pipeline and eventually storing it in the pickle file which is then saved in the folder.

For some reason it cannot detect the pickle file import and throws following error:

   Use a production WSGI server instead.
 * Debug mode: off
Traceback (most recent call last):
  File "C:\ANACONDA3\Scripts\flask-script.py", line 9, in <module>
    sys.exit(main())
  File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 967, in main
    cli.main(args=sys.argv[1:], prog_name="python -m flask" if as_module else None)
  File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "C:\ANACONDA3\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\ANACONDA3\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\ANACONDA3\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\ANACONDA3\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\ANACONDA3\lib\site-packages\click\decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\ANACONDA3\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 848, in run_command
    app = DispatchingApp(info.load_app, use_eager_loading=eager_loading)
  File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 305, in __init__
    self._load_unlocked()
  File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 330, in _load_unlocked
    self._app = rv = self.loader()
  File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 388, in load_app
    app = locate_app(self, import_name, name)
  File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 240, in locate_app
    __import__(module_name)
  File "C:\Users\bill-\OneDrive\Dokumente\Docs Bill\TA_files\functions_scripts_storage\flask_test\flask_regression_pipeline.py", line 18, in <module>
    @store_reg_pickle
NameError: name 'store_reg_pickle' is not defined

If you wish I could upload the entire scripts but that is a lot to look through and since it is working as a long regular pice of code, the mistake needs to be somewhere with my flask setup.

0 Answers
Related