I'm currently working on an ESP32CAM project to publish on AWS IOT topic some captures from the camera in high resolution (UXGA). I've managed to publish some short json payloads with attributes to different AWS IOT certificate protected topics but I'm facing an annoying issue to do the same with large payload as the capture binary file.
I've browsed many sites, forums, tested differents libs like MQTT, PubSubClient, AsyncMQTTClient... but I've not found a true working solution for large payload around 100KB size.
For example with the PubSubClient lib, I try to fragment my binary payload with the BeginPublish, write, endPublish scheme as below :
bool publishBinary(const uint8_t *buffer,size_t len, const char *topicPubName)
{
Serial.print("publishing binary ["+(String)len+"] ...");
if (len == 0) {
// Empty file
Serial.println("Error : binary payload is empty!");
return(false);
}
if (!client.beginPublish(topicPubName,len,false)) {
Serial.println("MQTT beginPublish failed.");
return(false);
}
size_t max_transfer_size=80;
size_t n=0;
size_t size_send;
size_t offset=0;
while ((len-offset)>0) {
n=(len-offset);
if (n > max_transfer_size)
n=max_transfer_size;
size_send=client.write((const uint8_t *)(buffer+offset),n);
Serial.printf("%d/%d : %.02f %%\n",offset,len,(double)((100*offset)/len));
//Serial.println("n: "+(String)n+" - send: "+(String)size_send);
if(size_send != n) {
// error handling. this is triggered on write fail.
Serial.println("Error during publishing..."+(String)size_send+" instead of "+(String)n);
client.endPublish();
return(false);
} else {
offset+=size_send;
}
}
client.endPublish();
Serial.println("ok");
return(true);
}
client is defined as PubSubClient client(net) where net is WiFiClientSecure object with validated CA_cert, cert and private key.
The MQTT connection is working well but when I try to publish the large binary payload, the function fragments buffer into chunks till the end but there is quite always an error like UNKNOWN ERROR CODE (0050) or when it succeeds to publish, only a small part of payload is published on the destination. In this case, my jpeg file is truncated on my S3 bucket where the payload lands.
I have to say that sometimes, I managed to publish a 65K payload but like a stroke of luck... :-)
I've looked for some examples on the web but very often it is for small payload. As mentioned in a post, I've tested the Publish_P(...) from PubSubClient... but same result, it aborts during transfer.
I begin to ask myself if it really possible by mqtt topic or do I have to create an API gateway with a lambda to handle such large payload. Tell me I'm wrong :-)
If you know a good solution for a true working large payload publishing, I would be delighted to discuss with you :-)
Thanks !