Wordpress - wrapping my binary file with double quotes

I have been trying to use my wordpress backend API to download a binary file(application/pdf) from external API.

Downloading directly from external API gives me correct results:

%PDF-1.4
%����
1 0 obj
<</Creator (Chromium)
...

But when I am calling it from wordpress instance and returning it as a response I am getting the same binary file, but wrapped with double quotes:

"%PDF-1.4\n%????\n1 0 obj\n<<\/Creator (Chromium)\n..."

Maybe this has something from encoding, but I don't know where to lookup.

Code responsible for calling external API for genereted binary file.

    public function execute(int $userId, int $moduleId): array
    {
        // ...
        $response = wp_remote_post('http://localhost:3100/pdf', [
            'headers' => [
                'Content-Type' => 'application/json',
            ],
            'body' => wp_json_encode([
                'user' => $user,
                'saved' => $saved,
                'settings' => ['acf' => $options],
                'wp' => $wp
            ]),
        ]);

        if(is_wp_error($response)) {
            throw new InternalBadRequestException('Unable to connect with pdf server.');
        }

        return $response;
    }

My Controller method

    public function generateSummary(WP_REST_Request $request, GenerateSummary $data)
    {
        // ...
        try {
            $summary = $data->execute(
                $this->getUserId(),
                $moduleId
            );
        } catch (ActionFailedException $e) {
            return new WP_Error(
                'store_error',
                $e->getMessage(),
                ['status' => 422]
            );
        }

        return new WP_REST_Response(
            $summary['body'],
            200,
            ['Content-Type' => 'application/pdf']
        );