Python HTTP Request Exception: LineTooLong

user_1140289 0
Viewed 1074

In pyhton, When I am trying to get json data from an URL I got the following exception.

('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line'))

But when I try curl it works without problem.

curl "http://ccmixter.org/api/query?f=json&datasource=default&lic=pd&sort=id&limit=20&offset=20"

Here is the sample python code which raise the exception:

import requests

url = "http://ccmixter.org/api/query?f=json&datasource=default&lic=pd&sort=id&limit=20&offset=20"
req = requests.get(url)
print(req.content)
python-requests httpclient python
2 Answers

This is caused by the http.client module, which has some limits on the header size.

You have two options :

  • You can manually change _MAXLINE to a higher value, knowing that the header size will not exceed it.

      import http.client

  http.client._MAXLINE = 655360

  url = "http://ccmixter.org/api/query?f=json&datasource=default&lic=pd&sort=id&limit=20&offset=20"
  req = requests.get(url)
  print(req.content)

  • If you don't want to worry about the header size, you can override the parse_headers function from the http.client module with a custom one in which I removed the part of the code that raises the exception once the limit is exceeded.

      def disable_header_limit():
      import http.client
      import email.parser
      from http.client import HTTPMessage, _MAXLINE

      def parse_headers(fp, _class=HTTPMessage):
          headers = []
          while True:
              line = fp.readline(_MAXLINE + 1)
              headers.append(line)
              if line in (b'\r\n', b'\n', b''):
                  break
          hstring = b''.join(headers).decode('iso-8859-1')
          return email.parser.Parser(_class=_class).parsestr(hstring)

      http.client.parse_headers = parse_headers

  disable_header_limit();
  url = "http://ccmixter.org/api/query?f=json&datasource=default&lic=pd&sort=id&limit=20&offset=20"
  req = requests.get(url)
  print(req.content)
user_45936420

This is caused by max limit set by http.client module. You can change it.

In [102]: import http.client

In [103]: http.client._MAXLINE = 655360

In [104]: res = requests.get("http://ccmixter.org/api/query?f=json&datasource=default&lic=pd&sort=id&limit=20&offset=20")

In [105]: res.json()
Out[105]:
[{'upload_id': 59180,
  'upload_name': 'trap & trance',
  'upload_extra': {'usertags': 'synthesizer,drums',
   'ccud': 'sample,media,bpm_100_105,how_i_did_it',
   'systags': 'cczero,audio,mp3,48k,stereo,VBR,audio,mp3,48k,stereo,VBR,audio,mp3,48k,stereo,VBR,audio,mp3,48k,stereo,VBR,audio,mp3,48k,stereo,VBR',
   'nsfw': False,
   'relative_dir': 'content/fourtrack',
   'bpm': 100,
   'featuring': '',
   'last_op': 'add',
   'howididit': {'tools': 'reaper, zynaddsubfx, lots of compressors and effects',
    'samples': '',
    'origial': '',
...
...
user_68496820
Related