Pandas - Reading Large CSV directly from URL

user_8992936 0
Viewed 1104

I'm new to python/data science in general, trying to understand why the below isn't working:

import pandas as pd
url = 'https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2017-06.csv'
trip_df = []
for chunk in pd.read_csv(url, chunksize=1000, nrows=10000):
    trip_df.append(chunk)
trip_df = pd.concat(trip_df, axis='rows')

It's returning a MemoryError, but I was under the impression that loading the file in chunks was a workaround for that. I'd prefer not to save the csv to my hard drive, plus I'm not entirely sure if that would help.

My computer's pretty limited, only 4GB of RAM (2.9 usable). I imagine that's a factor here, but if importing the whole file isn't an option, how would I just get part of it, say 5,000 rows?

pandas csv url python
3 Answers

Try this:

reader = pd.read_csv(url, chunksize=1000, nrows=10000)
df = pd.concat([x for x in reader], ignore_index=True)

how would I just get part of it, say 5,000 rows?

df = pd.read_csv(url, nrows=5000)
user_57412050

This works for me

with contextlib.closing(urllib.request.urlopen(url=url)) as rd:
    for df in pd.read_csv(rd, chunksize=5, index_col=0):
        print(df)

while just

pd.read_csv(url, chunksize=5, index_col=0)

crashes with MemoryError.

user_36095680

Just get the 5000 rows in the command line with

wget -q -O - https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2017-06.csv | head -n 5000 > tmp.csv

and then proceed with tmp.csv

user_33351630
Related