How can I read a tar.gz file directly from a URL into Pandas?

user_7472119 0
Viewed 1541

The dataset I wish to read lives on GitHub as a tar.gz file and is updated every few hours. While I can always download this file, uncompress it, and read from CSV, it would be much better if I can directly read from this URL into a Pandas data frame in a timely manner.

After some Googling, I was able to download the compressed file and then read it as a data frame.

import requests
import tarfile
import pandas as pd

# Download file from GitHub
url = "https://github.com/beoutbreakprepared/nCoV2019/blob/master/latest_data/latestdata.tar.gz?raw=true"
target_path = "latestdata.tar.gz"

response = requests.get(url, stream=True)
if response.status_code == 200:
    with open(target_path, "wb") as f:
        f.write(response.raw.read())

# Read from downloaded file
with tarfile.open(target_path, "r:*") as tar:
    csv_path = tar.getnames()[0]
    df = pd.read_csv(tar.extractfile(csv_path), header=0, sep=",")

However, I wonder if there is a way to directly read the file content into a data frame without first saving it locally. This may be useful if I want to build a web app later and don't have a local machine. Any help would be appreciated! Thanks!

data-wrangling python-requests pandas file python
2 Answers

You can use the BytesIO(In-Memory Stream) to keep the data in memory instead of saving the file to local machine.

Also As per the tarfile.open documentation, If fileobj is specified, it is used as an alternative to a file object opened in binary mode for name.

>>> import tarfile
>>> from io import BytesIO
>>>
>>> import requests
>>> import pandas as pd


>>> url = "https://github.com/beoutbreakprepared/nCoV2019/blob/master/latest_data/latestdata.tar.gz?raw=true"
>>> response = requests.get(url, stream=True)
>>> with tarfile.open(fileobj=BytesIO(response.raw.read()), mode="r:gz") as tar_file:
...     for member in tar_file.getmembers():
...         f = tar_file.extractfile(member)
...         df = pd.read_csv(f)
...         print(df)
user_66994470

If you use ParData, this can be done pretty cleanly:

from tempfile import TemporaryDirectory

import pardata

schema = {
    'download_url': 'https://github.com/beoutbreakprepared/nCoV2019/blob/master/latest_data/latestdata.tar.gz?raw=true',
    'subdatasets': {
        'all': {
            'path': 'latestdata.csv',
            'format': {
                'id': 'table/csv'
            }
        }
    }
}

with TemporaryDirectory() as d:
    dataset = pardata.dataset.Dataset(schema=schema, data_dir=d)
    dataset.download(verify_checksum=False)
    my_csv = dataset.load()  # my_csv is a pandas.DataFrame object that stores the CSV file

print(my_csv)

Disclaimer: I'm a primary co-maintainer of ParData.

user_11504620
Related