Uploading File to GCS - Metadata part is too large

user_7461941 0
Viewed 464

We are trying to upload files to GCS using signed urls. The client asks for a signed url to upload a file and our server sends a signed url to the client. The client then uploads the file using that signed url. This is working fine for cases where file size < 2MB. But the limit for the object size in GCS is 5TB.

Below is the code used to generate the signed url


blob = Blob.from_string(filepath, client=gcp_storage_client)
blob.generate_signed_url(version="v4", expiration=datetime.timedelta(minutes=5), method="POST")

We are getting the response "Metadata part is too large". But we are not adding any metadata to the file. Please check the screenshot from Postman below.

What am I doing wrong here? and how can I fix it?

enter image description here

pre-signed-url google-cloud-storage python
1 Answers

GCS has several mechanisms to upload objects. The one you're using is the HTML form "POST object" mechanism. POST uploads do not support signed URLs. They use signed policy documents instead, which are a bit more complicated but have some extra capabilities.

If your client is using not a human being using an HTML form, I suggest using PUT object for uploads. It's simpler, and it supports signed URLs. It would look like this:

PUT /OBJECT_NAME??X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=whatnot HTTP/1.1
Host: BUCKET_NAME.storage.googleapis.com
Date: DATE
Content-Length: FILE_LENGTH
Content-Type: FILE_MIME_TYPE
Content-MD5: MD5_DIGEST
x-goog-storage-class: DESIRED_STORAGE_CLASS

<file goes here>
user_814910
Related