Parsing multipart/mixed responses with Node.js and axios

Viewed 888

I've trying to parse a multipart/mixed response with (1) JSON and (2) .zip file.

I've been using Axios to GET the response, no problem there. I've seen npm packages like meros and multipart-mixed-parser but both just returned an empty array.

the response looks like

--Boundary_137_1895017408_1627074248456
Content-Type: application/json

{
  "createdDate" : "2021-07-22T01:46:05.149+0000",
   more JSON...
}

--Boundary_137_1895017408_1627074248456
Content-Type: application/octet-stream
Content-Disposition: form-data; filename="347b3bd1-e6e3-49a0-928d-9956df5c5af11671844264911435718.tmp"; modification-date="Fri, 23 Jul 2021 21:04:08 GMT"; size=620991; name="attachments.zip"

binary data here ...

--Boundary_137_1895017408_1627074248456--

    async function eManGet (mtn, zipPath) {
      try {
        mtn = mtn.toUpperCase()
        if (zipPath) {
          const res = eManAPI.get({
            url: `/emanifest/manifest/${mtn}/attachments`,
            headers: {
              Accept: 'multipart/mixed'
            }
          })
          return res
        } else {
         ...
        }
        // console.log(res.data)
      } catch (error) {
       ...
      }
    }

The package I want to use this for is here

If you've faced this problem before, what package(s) did you use?

2 Answers

The dicer npm package does the job pretty well. You'll have to extract the boudary from headers yourself tho.

How to send multipart/mixed with axios

There is example of source code with request

const messages = body[endpoint].map(function (m) {
    return {
        'Content-Type': 'application/http',
        body: 'GET ' + api + '/gmail/v1/users/me/' + endpoint + '/' + m.id + query + '\n'
    }
})

const r = request({
    method: 'POST',
    url: api + '/batch/gmail/v1',
    multipart: messages,
    timeout: opts.timeout,
    headers: {
        'Authorization': 'Bearer ' + key,
        'content-type': 'multipart/mixed'
    }
})

from this library

https://github.com/SpiderStrategies/node-gmail-api

To solve this problem I used request catcher and wrote the following tests:

it('request catcher of request', () => {
  const catcher = 'https://pl.requestcatcher.com/';

  const messages = [1,2,3].map(function (index:number) {
    return {
      'Content-Type': 'application/http',
      body: 'GET ' + `/gmail/v1/users/me/${index}\n`
    }
  })

  const r = request({
    method: 'POST',
    url: catcher,
    multipart: messages,
    headers: {
      'Authorization': 'Bearer KEY',
      'content-type': 'multipart/mixed'
    }
  })

  expect(r).toBeNull();
})

and saw the following request on catcher:

enter image description here

For axios there is not ready answer in all internet. This is my research:

You can use a FormData object in the browser. In Node.js the feature isn't supported yet by axios (see #789).

https://github.com/axios/axios/issues/803

[RFC2388] suggested that multiple files for a single form field be transmitted using a nested "multipart/mixed" part. This usage is deprecated.

https://datatracker.ietf.org/doc/html/rfc7578

info how to do it in request ( that is also depreciated )

https://github.com/request/request#multipartrelated

Function for generation boundary

https://github.com/form-data/form-data/blob/53adbd81e9bde27007b28083068f2fc8272614dc/lib/form_data.js#L347

So we can send multipart/mixed by this code

  it('request catcher of axios', () => {
    const catcher = 'https://pl.requestcatcher.com/';

    const r = axios.post(catcher, `--3e50fc2b-7a5c-4565-99f2-1628010018be
Content-Type: application/http

GET /gmail/v1/users/me/1

--3e50fc2b-7a5c-4565-99f2-1628010018be
Content-Type: application/http

GET /gmail/v1/users/me/2

--3e50fc2b-7a5c-4565-99f2-1628010018be
Content-Type: application/http

GET /gmail/v1/users/me/3

--3e50fc2b-7a5c-4565-99f2-1628010018be--`,{
      headers: {
        'Authorization': 'Bearer KEY',
        'content-type': 'multipart/mixed; boundary=3e50fc2b-7a5c-4565-99f2-1628010018be'
      }
    })

    expect(r).not.toBeNull();

  })

and now oly generation of boundary is problem to solve. Lets check differences between multipart/mixed and and multipart/form-data bounduaries.

This is exmple of bounduary from form-data package:

--------------------------942880057551350107765038

and this is from mixed (from request):

--3e50fc2b-7a5c-4565-99f2-1628010018be

and this is proper RFC

https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html

so we can use this code

> `--${crypto.randomUUID()}`
'--7784084f-e129-44ea-aa64-8f480a26c75f'

And this is code that sends mutipart/mixed by axios

    it('request catcher of axios', () => {
        const catcher = 'https://pl.requestcatcher.com/';

        interface SingleMultipartMixedMessage {
            'Content-Type': string,
            body: string
        }

        const messages = [1, 2, 3].map(function (index: number) {
            return {
                'Content-Type': 'application/http',
                body: 'GET ' + `/gmail/v1/users/me/${index}\n`
            }
        })

        const boundary = `--${crypto.randomUUID()}`

        const body = messages.reduce((p: string, n: SingleMultipartMixedMessage) => {
            return p + `\nContent-Type: ${n['Content-Type']}\n\n${n.body}\n${boundary}`
        }, boundary)

        const r = axios.post(catcher, body, {
            headers: {
                'Authorization': 'Bearer KEY',
                'content-type': `multipart/mixed; boundary=${boundary.replace(/^--/, '')}`
            }
        })

        expect(r).not.toBeNull();

    })

and comparison of requests caught by caught between reference implementation [ request ] and our code [ axios ]

enter image description here

How to parse multipart mixed

  1. Get boundary from content-type header
  2. Add -- before it
  3. Split body by this boundary
  4. Any part split by double \n
  5. Parse any pair of headers and data individually
Related