My docker-composed setup consists of 3 containers - mlflow + minio(s3) + mysql and I am trying to run a simple sklearn training model python script from within the mlflow container and it works fine but for the last line below -
import pandas as pd
import mlflow
import mlflow.sklearn
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("test1")
df = pd.read_csv("kc_house_data.csv")
# choose features
features = ["bedrooms","bathrooms","sqft_living","sqft_above","grade",
"floors","view",'sqft_lot','floors','waterfront','zipcode']
# getting those features from the dataframe
x = df[features]
y = df["price"]
# splits data into 80% train 20% test
x_train, x_test, y_train, y_test = train_test_split(x, y, train_size=0.7,
random_state=3)
# choose model with settings
model = RandomForestRegressor(n_estimators=100)
model.fit(x_train, y_train)
# define and print
metrics = {"train_score": model.score(x_train, y_train),
"test_score": model.score(x_test, y_test)}
print(metrics)
# log params to mlflow and artifacts to minio
mlflow.log_params({"n_estimators":100})
mlflow.log_metrics(metrics)
mlflow.sklearn.log_model(model, "foobar-model")
However, when I try to log the model (last line ^) to a remote minio (s3) dockerized server, I get the following error -
botocore.exceptions.ClientError: An error occurred (InternalError) when calling the UploadPart operation (reached max retries: 4): We encountered an internal error, please try again.: cause(open /data/.minio.sys/multipart/f96f6e560d343e6f7dedc59b80e14aa631577530b9c8cbe55fddb9d7dc67369e/77497a07-cf9e-4c1b-a444-7cf78e03de07/fs.json: invalid argument)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train.py", line 36, in <module>
mlflow.sklearn.log_model(model, "foobar-model")
File "/usr/local/lib/python3.7/site-packages/mlflow/sklearn/__init__.py", line 341, in log_model
extra_pip_requirements=extra_pip_requirements,
File "/usr/local/lib/python3.7/site-packages/mlflow/models/model.py", line 188, in log
mlflow.tracking.fluent.log_artifacts(local_path, artifact_path)
File "/usr/local/lib/python3.7/site-packages/mlflow/tracking/fluent.py", line 584, in log_artifacts
MlflowClient().log_artifacts(run_id, local_dir, artifact_path)
File "/usr/local/lib/python3.7/site-packages/mlflow/tracking/client.py", line 977, in log_artifacts
self._tracking_client.log_artifacts(run_id, local_dir, artifact_path)
File "/usr/local/lib/python3.7/site-packages/mlflow/tracking/_tracking_service/client.py", line 334, in log_artifacts
self._get_artifact_repo(run_id).log_artifacts(local_dir, artifact_path)
File "/usr/local/lib/python3.7/site-packages/mlflow/store/artifact/s3_artifact_repo.py", line 108, in log_artifacts
key=posixpath.join(upload_path, f),
File "/usr/local/lib/python3.7/site-packages/mlflow/store/artifact/s3_artifact_repo.py", line 80, in _upload_file
s3_client.upload_file(Filename=local_file, Bucket=bucket, Key=key, ExtraArgs=extra_args)
File "/usr/local/lib/python3.7/site-packages/boto3/s3/inject.py", line 132, in upload_file
extra_args=ExtraArgs, callback=Callback)
File "/usr/local/lib/python3.7/site-packages/boto3/s3/transfer.py", line 287, in upload_file
filename, '/'.join([bucket, key]), e))
boto3.exceptions.S3UploadFailedError: Failed to upload /tmp/tmprf9bfrwg/model/model.pkl to foobar/1/9a0a9b37750040e8b8725cf9088a0209/artifacts/foobar-model/model.pkl: An error occurred (InternalError) when calling the UploadPart operation (reached max retries: 4): We encountered an internal error, please try again.: cause(open /data/.minio.sys/multipart/f96f6e560d343e6f7dedc59b80e14aa631577530b9c8cbe55fddb9d7dc67369e/77497a07-cf9e-4c1b-a444-7cf78e03de07/fs.json: invalid argument)
The minio (s3) logs show -
API: PutObjectPart(bucket=foobar, object=1/9a0a9b37750040e8b8725cf9088a0209/artifacts/foobar-model/model.pkl)
Time: 04:47:51 UTC 10/23/2021
DeploymentID: 29aa997e-f56b-4697-aa66-84417aa00646
RequestID: 16B08F8654D8CB84
RemoteHost: 172.24.0.4
Host: s3:9000
UserAgent: Boto3/1.19.1 Python/3.7.12 Linux/5.10.25-linuxkit Botocore/1.22.1
Error: open /data/.minio.sys/multipart/f96f6e560d343e6f7dedc59b80e14aa631577530b9c8cbe55fddb9d7dc67369e/77497a07-cf9e-4c1b-a444-7cf78e03de07/fs.json: invalid argument (*fs.PathError)
4: cmd/api-errors.go:2082:cmd.toAPIErrorCode()
3: cmd/api-errors.go:2107:cmd.toAPIError()
2: cmd/object-handlers.go:2764:cmd.objectAPIHandlers.PutObjectPartHandler()
1: net/http/server.go:2046:http.HandlerFunc.ServeHTTP()
BTW, if I exec into my mlflow server, I can connect to minio s3 container and issue aws cli commands to the required s3 buckets (aws s3 ls and aws s3 cp) using for example aws --endpoint-url http://s3:9000 s3 ls etc.
Can someone please help me troubleshoot my failure to log my sklearn model to minio (s3) ?
TIA.