Is it possible to download and POST with multipart/form-data same file simultaneously, with libcurl?

Viewed 390

Maybe this is a dumb question, given that I'm experimenting with libcurl, and also could be the case that is not possible to do. Let me describe you first the behavior that I want to achieve.

Suppose that I want to download some file, I could request this file by pieces until I get the whole file. Now, instead of storing this data into a temporary file, every time I got one of these pieces or chunk, I want to upload this piece to another server. Both servers are HTTP server, and the second one only accept the upload of the file, using a POST with multipart/form-data. I have an idea, but I couldn't make it work, and still not sure if it's possible. The idea that I had was to mimic the solution that I found in this question, but here they are doing it with the command line curl and the second server is an FTTP server.

So the idea is to create a pipe, and use this pipe to pass the pieces to the part that is responsible for the upload. Let's show you a sketch, in case you got lost in my poor English: data flow

Maybe the pipe is not needed at all, because libcurl allows you to access these chunks of data, that you are downloading, like in the getinmemory example, but I don't know if it is possible to pass these pieces into a one multipart/form-data request, without storing the data into a file and later telling to libcurl to upload that file.

As I said before, I try to make this solution of the pipe work, but couldn't make it work. Let me show my incomplete code, so you get a concrete idea:

#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/wait.h>
#include <curl/curl.h>

#define WE 1
#define RE 0



/*down_up video*/
CURLcode down_up(void)
{
  CURLcode ret;
  const char *url_down = "http://localhost:4000/static/videos/1.mp4";
  const char *url_up = "http://localhost:4000/upload";

  int pdf[2]; /*write end and read end*/

  if(pipe(pdf) == -1)
    errExit("pipe");

  pid_t pid = fork();
  switch(pid) {
  case -1:
    errExit("ok");
  case 0:
    /*closing unused WE in child proccess*/
    if(close(pdf[WE]) == -1)
      errExit("Close");
    break;
  default:
    /*parent process*/
    /*closing the RE on parent proccess*/
    if(close(pdf[RE]) == -1)
      errExit("Close");
    break;
  }

  switch(pid) {
  case -1:
    errExit("ok");
  case 0:
    /*child process*/
    CURL *curl_up;
    curl_up = curl_easy_init();
    curl_easy_setopt(curl_up, CURLOPT_URL, url_up);
   /*upload file with multipart/form-data, reading chunks from pipe*/

    break;
  default:
    /*parent process*/
    CURL *curl_down;
    curl_down = curl_easy_init();
    curl_easy_setopt(curl_down, CURLOPT_URL, url_down);
    /*download file, and writing pieces into the pipe*/

    break;
  }


  curl_easy_cleanup(curl_down);
  curl_easy_cleanup(curl_up);
  curl_down = NULL;
  curl_up = NULL;
  return ret;
}

int main(void)
{
  CURLcode ret;
  ret = download_upload();
  if(ret != CURLE_OK)
    return 1;
  return 0;
}

Here I'm trying to download with the parent process, and upload with the child process. In order to pass data from the parent to the child process, I close the Read end(RE) of the PIPE in the parent process, and close also the Write end(WE) of the PIPE in the child process. We get something like this:

pipe after fork

But still don't know how to orchestrate all this. Is this even possible?

1 Answers

Yes it is certainly possible.

The basic functionality could be made to work like this. Two threads:

  1. The downloading of the file stores it in memory, more or less appending to a buffer. Let's call it the "download buffer". A real-life implementation would possibly pause the transfer if the download buffer grows too big. Ie if the uploading side is much slower than the downloader.

  2. The uploading of the file in a multipart formpost is done in a separate thread and uses the libcurl's CURLOPT_MIMEPOST option to create the post body. The part that uploads the file from memory is created with curl_mime_data_cb, so that a callback function you provide is called when libcurl wants data to upload for that part. Your callback would then provide data from the "download buffer" mention in step 1. If there's no data available, the callback probably just waits until there is data before it returns.

Related