python 3: urllib, csv reader, and StringIO- why the difference?

user_160511 0
Viewed 437

These two approaches yield different results in python 3.7.3:

res = urllib.request.urlopen(url, timeout=timeout)
content = res.read().decode('utf-8')
reader = csv.reader(StringIO(content))
lines = list(reader)

And

res = urllib.request.urlopen(url, timeout=timeout)
content = res.read().decode('utf-8')
reader = csv.reader(content)
lines = list(reader)

The former gives me what I want, a list of the rows from the CSV, the latter gives me a list containing lists of length 1 of single characters only (each character in the text is its own list), so:

Year,PID
2019,1
2018,2

And

Y
e
a
r,
P
i
d
(etc)

What's the difference?

urllib python-3.x python
2 Answers

in-memory stream for text I/O

  • It's important to note that:

For strings StringIO can be used like a file opened in text mode

csv.reader treats StringIO(content) as open file. And reader is

a reader object which will iterate over lines in the given csvfile

  • So lines = list(reader) will return you a list of lines in content

In the second case content is of type string.

  • So csv.reader(content) will return an iterator over the string.

And this is because:

csv.reader(csvfile, dialect='excel', **fmtparams) csvfile can be any object which supports the iterator protocol and returns a string each time its next() method is called

  • That's why lines = list(reader) returns a list of characters, as it treats each character in content as a row.
user_12643040

Have a look at the documentation for csv.reader:

Return a reader object which will iterate over lines in the given csvfile. csvfile can be any object which supports the iterator protocol and returns a string each time its __next__() method is called — file objects and list objects are both suitable.

If you're not sure what "the iterator protocol" is, it's basically equivalent to what happens when you loop over some object in a for loop. For example with a list:

>>> l = [1, 2, 3]
>>> for item in l:
...     print(item)
1
2
3

That is, for lists the iterator protocol is implemented in the "obvious" way that it returns each element of the list in order. For strings:

>>> s = 'row,from,csv'
>>> for char in s:
...     print(char)
...
r
o
w
,
f
r
o
m
,
c
s
v

For most "file-like" objects like StringIO the iterator protocol is defined such that each loop returns a single line of text (with the trailing newline):

>>> s = """\
... row,1
... row,2
... """
>>> b = StringIO(s)
>>> for line in b:
...     print(line)
...
row,1

row,2

So in the latter case it's treating a simple string as a sequence of one character "rows" from the CSV.

user_9822570
Related