Determining binary data vs. text using the HTTP Content-Type header

user_7361479 0
Viewed 1040

I'm writing code to read arbitrary web resources via HTTP/HTTPS in a Node.js environment, and I'd like to return the content as a string when it's text data, and as a Buffer when it's binary data.

It's clear that anything that starts with text, like text/html, is text data best returned as a string — using the appropriate character encoding, of course, which may be defined explicitly (such as text/html; charset=utf-8), or may not be. Further, any explicit definition of charset would indicate that the content is text, not binary, regardless of the MIME type.

As far as I can tell, almost everything else is binary data. All audio and video formats that I know of are binary, and nearly every image type, with the exception of image/svg+xml. It seems that most application/... types are binary, but with some notable common exceptions like application/json.

Does the following function seem to adequately determine the issue? If not, what notable exceptions might I be missing?

function isBinary(contentType: string): boolean {
  let $: string[];

  if (/;\s*charset\s*=/i.test(contentType))
    return false;

  // Remove anything other than MIME type.
  contentType = contentType.replace(/;.*$/, '').trim();

  if (/^text\//i.test(contentType) || /\+xml$/i.test(contentType))
    return false;
  else if (($ = /^application\/(.+)/i.exec(contentType)))
    return !/^(javascript|ecmascript|json|ld\+json|rtf)$/i.test($[1]);
  else
    return true;
}
typescript content-type mime-types http
1 Answers

Well you can istextorbinary on the buffer you fetch from the URL.

For example in lambda you would:

const fetch = require('node-fetch');
const { isText, isBinary, getEncoding } = require('istextorbinary');
module.exports.handler = async (event, context, callback) => {
.
.
        const customUrl = 'www.abc.net.au';
        const url = `https://${customUrl}${event.path}`;

        // Request with GET/HEAD method cannot have body | undefined
        // Set fetch parameters in params
        var params = {};
        if (event.httpMethod === 'GET' || event.httpMethod === 'HEAD') {
            params = {
                method: event.httpMethod,
                headers: customRequestHeader
            };
        } else {
            params = {
                method: event.httpMethod,
                headers: customRequestHeader,
                body: JSON.stringify(parsedbody)
            };
        }

        console.debug('request params: ' + JSON.stringify(params));

        // Fetch URL with params
        const response = await fetch(url, params);
        var textResponse = await response.buffer();

        var isBase64EncodedValue = false;  
        if ( isBinary(null, textResponse) ) {
            console.log('textResponse is a binary blob - setting isBase64Encoded to true on returnResponse');
            isBase64EncodedValue = true;
            console.log('isBase64EncodedValue in returnResponse is: ' + isBase64EncodedValue);
            // We need to return binary base64 encoded data for binary files - otherwise APIGW will throw 500 error back to the user
            textResponse = textResponse.toString('base64');
            // If you need to convert it back you could do something like - textResponse.toString('binary')
            console.log('When isBase64EncodedValue is true, textResponse is: ' + textResponse);
        } else {
            console.log('textResponse is not a binary blob - setting isBase64Encoded to false on returnResponse');
            isBase64EncodedValue = false;
            console.log('isBase64EncodedValue in returnResponse is: ' + isBase64EncodedValue);
            textResponse = textResponse.toString('utf8');
            console.log('When isBase64EncodedValue is false, textResponse is: ' + textResponse);
        }

.
.
};

Did you end up using your function? Wanna share your final code?

user_128302570
Related