gzip compress output of an aws python lambda (in code)

I have an aws python lambda that returns a PDF file (generated via pyppeteer). It works fine, except that sometimes the PDF is larger than 6MB and then the lambda fails.

This is the return statement that fails with large files, based on this:

    return {
        'headers': {"Content-Type": "application/pdf"},
        'statusCode': 200,
        'body': base64.b64encode(pdf).decode('utf-8'),
        'isBase64Encoded': True
    }

Hoping to compress the output, I wrote the following code:

    return {
        'headers': {
            "Content-Type": "application/pdf",
            "Content-Encoding": "gzip",
        },
        'statusCode': 200,
        'body': base64.b64encode(gzip.compress(pdf)).decode('utf-8'),
        'isBase64Encoded': True
    }

The output is quite odd - using insomnia I can see that the output starts with what seems like a valid PDF, but the output's size is smaller than it would be before this change, and the output is no longer a valid PDF.

Note: compression is configured in serverless.yml Tried turning this off, but it didn't help.

Is there a way to gzip compress the lambda's output via code? (I know it'll solve just part of the problem, as it'll still be possible to create a PDF that's compressed and larger than 6MB, but it'll definitely reduce the occurrence of this issue)