How to handle a .csv input for use in Tensorflow Serving batch transform?

Information: I am loading an existing trained model.tar.gz from an S3 bucket, and want to perform a batch transform with a .csv containing the input data. The data.csv is structured in such a way that reading it into a pandas DataFrame gives me rows of complete prediction inputs.

• This is done on Amazon Sagemaker using the Python SDK
• BATCH_TRANSFORM_INPUT is the path to data.csv.
• I'm able to load the contents inside model.tar.gz and use them for inference on my local machine using tensorflow, and the logs show 2020-08-04 13:35:01.123557: I tensorflow_serving/core/loader_harness.cc:87] Successfully loaded servable version {name: model version: 1}so the model seems to have been trained and saved properly.
• The data.csv is in the exact same format as the training data, which means one row per "prediction" where all columns in that row represents the different features.
• Changing the argument strategy to 'MultiRecord' gives the same error
• [path in s3] is a substitute for the real path as i don't want to reveal any bucket information.
• TensorFlow ModelServer: 2.0.0+dev.sha.ab786af
• TensorFlow Library: 2.0.2

Where 1-5 are features, the file data.csv looks like:

+------+-------------------------+---------+----------+---------+----------+----------+
| UNIT | TS                      | 1       | 2        | 3       | 4        | 5        |
+------+-------------------------+---------+----------+---------+----------+----------+
| 110  | 2018-01-01 00:01:00.000 | 1.81766 | 0.178043 | 1.33607 | 25.42162 | 12.85445 |
+------+-------------------------+---------+----------+---------+----------+----------+
| 110  | 2018-01-01 00:02:00.000 | 1.81673 | 0.178168 | 1.30159 | 25.48204 | 12.87305 |
+------+-------------------------+---------+----------+---------+----------+----------+
| 110  | 2018-01-01 00:03:00.000 | 1.8155  | 0.176242 | 1.38399 | 25.35309 | 12.47222 |
+------+-------------------------+---------+----------+---------+----------+----------+
| 110  | 2018-01-01 00:04:00.000 | 1.81530 | 0.176398 | 1.39781 | 25.18216 | 12.16837 |
+------+-------------------------+---------+----------+---------+----------+----------+
| 110  | 2018-01-01 00:05:00.000 | 1.81505 | 0.151682 | 1.38451 | 25.22351 | 12.41623 |
+------+-------------------------+---------+----------+---------+----------+----------+

inference.py currently looks like:

def input_handler(data, context):
    import pandas as pd
    if context.request_content_type == 'text/csv':
        payload = pd.read_csv(data)
        instance = [{"dataset": payload}]
        return json.dumps({"instances": instance})
    else:
        _return_error(416, 'Unsupported content type "{}"'.format(context.request_content_type or 'Unknown'))

The problem:

When the following code runs in my jupyter Notebook:

sagemaker_model = Model(model_data = '[path in s3]/savedmodel/model.tar.gz'),  
                        sagemaker_session=sagemaker_session,
                        role = role,
                        framework_version='2.0',
                        entry_point = os.path.join('training', 'inference.py')
                        )

tf_serving_transformer = sagemaker_model.transformer(instance_count=1,
                                                     instance_type='ml.p2.xlarge',
                                                     max_payload=1,
                                                     output_path=BATCH_TRANSFORM_OUTPUT_DIR,
                                                     strategy='SingleRecord')


tf_serving_transformer.transform(data=BATCH_TRANSFORM_INPUT, data_type='S3Prefix', content_type='text/csv')
tf_serving_transformer.wait()

The model seems to get loaded, but I end up with the following error: 2020-08-04T09:54:27.415:[sagemaker logs]: MaxConcurrentTransforms=1, MaxPayloadInMB=1, BatchStrategy=SINGLE_RECORD 2020-08-04T09:54:27.503:[sagemaker logs]: [path in s3]/data.csv: ClientError: 400 2020-08-04T09:54:27.503:[sagemaker logs]: [path in s3]/data.csv: 2020-08-04T09:54:27.503:[sagemaker logs]: [path in s3]/data.csv: Message: 2020-08-04T09:54:27.503:[sagemaker logs]: [path in s3]/data.csv: { "error": "Failed to process element: 0 of 'instances' list. Error: Invalid argument: JSON Value: \"\" Type: String is not of expected type: float" }

Error more clearly:

ClientError: 400 Message: {"error": "Failed to process element: 0 of 'instances' list. Error: Invalid argument: JSON Value: "" Type: String is not of expected type: float"}

If i understand this error correctly, something is wrong with the way my data is structured, so that sagemaker fails to deliver the input data to the TFS model. I suppose there is some "input handling" missing in my inference.py. Maybe the csv data has to somehow be translated into a compatible JSON, for TFS to use it? What exactly has to be done in input_handler() ?

I appreciate all help, and am sorry for this confusing case. If there is any additional information needed, please ask and I'll gladly provide what I can.