Pandas - populate list with dataframe values as strings

Admin 1
Viewed 591

I am reading csv files from a folder and filtering tem into a pandas dataframe, like so:

results=[]
for filename in glob.glob(os.path.join('/path/*.csv')):
  with open(filename) as p:
    df = pd.read_csv(p)

    filtered = df[(df['duration'] > low1) & (df['duration'] < high1)]

    artist = filtered['artist'].values
    print artist
    track = filtered['track'].values
    print track

where low1 = 0, high_1 = 0.5

artist and track print hundreds of filtered items as normal strings, but if I try to append them to results in the loop:

artist = filtered['artist'].values
track = filtered['track'].values
results.append([track,artist])

I see that I am appendding objects and types and results ends up populated with a fraction of the filtered items. I don't get what happens.

How do I populate results with all items as regular strings, in this fashion:

[['artist1', 'track1'], ['artist1', 'track2], ...]]
pandas csv python
1 Answers

Create list of DataFrames and then join them together by concat, last convert to nested lists:

results=[]
for filename in glob.glob(os.path.join('/path/*.csv')):
    df = pd.read_csv(filename)
    #filter by conditions and also columns by names with .loc
    filtered = df.loc[(df['duration'] > low1) & (df['duration'] < high1), ['artist','track']]
    #alternative solution 
    filtered = df.loc[df['duration'].between(low1, high1,inclusive=False), ['artist','track']]
    results.append(filtered) 

out = pd.concat(results).values.tolist()

Another solution id append lists and last flattening them by list comprehension:

results=[]
for filename in glob.glob(os.path.join('/path/*.csv')):
    df = pd.read_csv(filename)
    #filter by conditions and also columns by names with .loc
    mask = df['duration'].between(low1, high1,inclusive=False)
    filtered = df.loc[mask, ['artist','track']].values.tolist()
    results.append(filtered) 

out = [y for x in results for y in x]
user_29010020
Related