How do I convert a list containing lines(in string) in a pandas data frame from an url using requests

user_18304650 0
Viewed 44

I'm trying to get a final pandas data frame from an initial uniprot url:

import requests
url = 'http://www.uniprot.org/uniprot/?query=Interferon+lambda+receptor+1&sort=score&format=tab'
req = requests.get(url)

The output of:

req.text

is something like this:

""Entry\tEntry name\tStatus\tProtein names\tGene names\tOrganism\tLength\nQ8IU57\tINLR1_HUMAN\treviewed\tInterferon lambda receptor 1 (IFN-lambda receptor 1) (IFN-lambda-R1) (Cytokine receptor class-II member 12) (Cytokine receptor family 2 member 12) (CRF2-12) (Interleukin-28 receptor subunit alpha) (IL-28 receptor subunit alpha) (IL-28R-alpha) (IL-28RA) (Likely interleukin or cytokine receptor 2).....

To get the lines I did:

lines = req.text.splitlines()

#every line is separated by a comma ' ', ' ', ... #columns are separated by '\t'

If I use:

import re
re.split(r'\t+', lines[0])

this gives to the correct splitting of each columns.

Out:
['Entry',
 'Entry name',
 'Status',
 'Protein names',
 'Gene names',
 'Organism',
 'Length']

However, if I want to do a for loop and do it on all lines I get an error: string2list(lines): list indices must be integers or slices, not str'

import re

def string2list(file):
    list = []
    for i in lines:
        re.split(r'\t+', lines[i])
        list += lines
        return list

My aim is to get a list of lists to finally use this code:

import pandas as pd
list_name = lines
df = pd.DataFrame (list_name, columns = lines[i])

Any ideas on what is the best approach? Is the conversion of string in a list to a list of list possible? what is the best way? Or is there another way to reach the pandas data frame directly from the url? Thank you in advance!

python-requests pandas dataframe url python
2 Answers

The simplest way to load that file into a DataFrame is to use pd.read_csv(), which supports url input.

import pandas as pd
url = 'http://www.uniprot.org/uniprot/?query=Interferon+lambda+receptor+1&sort=score&format=tab'
df = pd.read_csv(url, sep='\t')

BTW, Regarding your code:

def string2list(file):
    list = []
    for i in lines:
        re.split(r'\t+', lines[i])
        list += lines
        return list

There are several problems.

  • file is unused
  • lines is undefined
  • i is a string, not an integer.
  • Therefore, you probably meant re.split(r'\t+', i)
  • list += lines is probably not what you meant...
  • Your return statement is inside the for loop, rather than in the outer scope.

IIUC, I think you were aiming to write something like this:

def split_lines(file):
    with open(file, 'r') as f:
        lines = f.readlines()

    results = []
    for line in lines:
        words = re.split(r'\t', line.strip())
        results.append(words)
    return results
user_1620940

Thank you very much Stuart for the `pd.read_csv()' function. It does exactly what I needed in a very efficient way!

For the other for loop, thank you so much for correcting!

This worked too, thanks for your input on the for loop:

def split_lines2(url):
    req = requests.get(url) 
    lines = req.text.splitlines()

    results = []
    for line in lines:
        words = re.split(r'\t', line.strip())
        results.append(words)
    return results

test_x = split_lines2(my_url) #this give a list of lists
df = pd.DataFrame (data = test_x, columns = test_x[0])
df_drop_row_1 = df.drop(df.index[0])

Thanks again:)

user_183046500
Related