How to use pandas to check for list of values from a csv spread sheet while filtering out certain keywords?

Kream123 0
Viewed 92

Hey guys this is my first post. I am planning on building an anime recommendation engine using python. I came across a problem where I made a list called genre_list which stores the genres that I want to filter from the huge data spreadsheet I was given. I am using the Pandas library and it has an isin() function to check if the values of a list is included in the datasheet and its supposed to filter it out. I am using the function but its not able to detect "Action" from the datasheet although it is there. I got a feeling there's something wrong with the data types and I probably have to work around it somehow but I'm not sure how.

I downloaded my csv file from this link for anyone interested! https://www.kaggle.com/datasets/marlesson/myanimelist-dataset-animes-profiles-reviews?resource=download

import pandas as pd

df = pd.read_csv('animes.csv')

genre = True
genre_list = []

while genre:
    genre_input = input("What genres would you like to watch?, input \"done\" when done listing!\n")
    if genre_input == "done":
        genre = False
    else:
        genre_list.append(genre_input)
print(genre_list)
df_genre = df[df["genre"].isin(genre_list)]
# df_genre = df["genre"]
print(df_genre)

Outout: [1]: https://i.stack.imgur.com/XZzcc.png

pandas python
4 Answers

You want to check if ANY value in your user input list is in each of the list values in the "genre" column. The "isin" function will check if your input in it's entirety is in a cell value, which is not what you want here. Change that line to this:

df_genre = df[df['genre'].apply(lambda x: any([i in x for i in genre_list]))]

Let me know if you need any more help.

Chowlett20 
import pandas as pd

df = pd.read_csv('animes.csv')

genre = True
genre_list = []

while genre:
    genre_input = input("What genres would you like to watch?, input \"done\" when done listing!\n")
    if genre_input == "done":
        genre = False
    else:
        genre_list.append(genre_input)

# List of all cells and their genre put into a list
col_list = df["genre"].values.tolist()
temp_list = []

# Each val in the list is compared with the genre_list to see if there is a match
for index, val in enumerate(col_list):
    if all(x in val for x in genre_list):
        # If there is a match, the UID of that cell is added to a temp_list
        temp_list.append(df['uid'].iloc[index])
print(temp_list)

# This checks if the UID is contained in the temp_list of UIDs that have these genres
df_genre = df["uid"].isin(temp_list)
new_df = df.loc[df_genre, "title"]
# Prints all Anime with the specified genres
print(new_df)

This is another approach I took and works as well. Thanks for all the help :D

Kream1230

To make a selection from a dataframe, you can write this:

df_genre = df.loc[df['genre'].isin(genre_list)]
Сергей Кох0

I've downloaded the file animes.csv from Kaggle and read it into a dataframe. What I found is that the column genre actually contains strings (of lists), not lists. So one way to get what you want would be:

...
m = df["genre"].str.contains(r"'(?:" + "|".join(genre_list) + r")'")
df_genre = df[m]

Result for genre_list = ["Action"]:

         uid  ...                                               link
3       5114  ...  https://myanimelist.net/anime/5114/Fullmetal_A...
4      31758  ...  https://myanimelist.net/anime/31758/Kizumonoga...
5      37510  ...  https://myanimelist.net/anime/37510/Mob_Psycho...
7      38000  ...  https://myanimelist.net/anime/38000/Kimetsu_no...
9       2904  ...  https://myanimelist.net/anime/2904/Code_Geass_...
...      ...  ...                                                ...
19301  10350  ...  https://myanimelist.net/anime/10350/Hakuouki_S...
19303   1293  ...  https://myanimelist.net/anime/1293/Urusei_Yatsura
19304    150  ...  https://myanimelist.net/anime/150/Blood_
19305   4177  ...  https://myanimelist.net/anime/4177/Bounen_no_X...
19309    450  ...  https://myanimelist.net/anime/450/InuYasha_Mov...

[4215 rows x 12 columns]

If you want to transform the values of the genre column for some reason into lists, then you could do either

df["genre"] = df["genre"].str[1:-1].str.replace("'", "").str.split(r",\s*")

or

df["genre"] = df["genre"].map(eval)

Afterwards

df_genre = df[~df["genre"].map(set(genre_list).isdisjoint)]

would give you the filtered dataframe.

Timus0
Related