My TF Serving reads the models from Ceph via its S3 endpoint. Our internal endpoint has changed (also changed from HTTP to HTTPS). Starting TF Serving with the new endpoint doesn't cause any issues, i.e. it connects fine and reads the models. But when a new model is available, TF Serving suddenly crashes. This doesn't happen if I keep using our old endpoint.
I added logs to see if there's some issues with the S3 file system, i.e. I set:
TF_CPP_VMODULE: "s3_file_system=2,file_system_storage_path_source=2"
AWS_LOG_LEVEL: "trace"
But they don't seem to be helpful.
This is part of healthy logs:
...
2022-09-13 09:27:06.873782: I external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:84] Returning connection handle 0x7fd33800ba30
2022-09-13 09:27:06.873792: I external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:84] Obtained connection handle 0x7fd33800ba30
2022-09-13 09:27:06.910123: I external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:84] HTTP/1.1 200 OK
2022-09-13 09:27:06.910509: I external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:84] Transfer-Encoding: chunked
...
And when the issue occurs, the logs look like this:
...
2022-09-13 02:19:31.167630: I external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:84] Returning connection handle 0x560c7d611e00
2022-09-13 02:19:31.167638: I external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:84] Obtained connection handle 0x560c7d611e00 <--- same as before
2022-09-13 02:19:31.697364: I external/org_tensorflow/tensorflow/core/platform/s3/s3_file_system.cc:712] Stat on path: s3://tfserving-config/model/prod/empty.config <--- here the container restarted
...
Tested on TensorFlow Serving Docker images with version: 2.5.3 and 2.6.5.
Are there some ways to further debug this or get more helpful logs?
Setting TF_CPP_MAX_VLOG_LEVEL creates too many logs to be helpful.