Pandas isin returns an error when encountering an empty DF after using concat

user_6301394 0
Viewed 949

When using concat and then isin to drop all rows, I encounter:

ValueError: cannot compute isin with a duplicate axis.

Meanwhile if there are any rows left in the DF, there are no issues. Also, not using concat works in any case, returning an empty DF gracefully. I'm attempting to use concat as it's slightly faster in my use case.

Pandas 0.24.2 (latest).

df = pd.DataFrame()
for r in range(5):
    df = df.append({'type':'teine', 'id':r}, ignore_index=True)

# Problem line
df = pd.concat([df.reset_index(drop=True), pd.DataFrame({'type':'teine', 'id':5}, index=[0])], sort=True)
dfCopy = df.copy()
df.query("(type == 'teine')", inplace=True)
df = dfCopy[~dfCopy.isin(df).all(axis=1)]
python-3.6 pandas python
2 Answers

you can just avoid the duplicated index problem not duplicating it:

df = pd.DataFrame()
for r in range(5):
    df = df.append({'type':'teine', 'id':r}, ignore_index=True)

# Problem line
df = pd.concat([df.reset_index(drop=True), pd.DataFrame({'type':'teine', 'id':5}, index=[max(df.index.values)+1])], sort=True)
dfCopy = df.copy()
df.query("(type == 'teine')", inplace=True)
df = dfCopy[~dfCopy.isin(df).all(axis=1)]

Or better yet, you can reset your index after concatenate:

df = pd.DataFrame()
for r in range(5):
    df = df.append({'type':'teine', 'id':r}, ignore_index=True)

# Problem line
df = pd.concat([df, pd.DataFrame({'type':'teine', 'id':5}, index=[0])], sort=True).reset_index(drop=True)
dfCopy = df.copy()
df.query("(type == 'teine')", inplace=True)
df = dfCopy[~dfCopy.isin(df).all(axis=1)]
user_115694230

In the line :

df = pd.concat([df.reset_index(drop=True), pd.DataFrame({'type':'teine', 'id':5}, index=[0])], sort=True)

there is no need to reset the index inside the concat (df.reset_index(drop=True)). But you have to reset the index after the concat to avoid your error. Here is what it looks like :

df = pd.concat([df, pd.DataFrame({'type':'teine', 'id':5}, index=[0])], sort=True).reset_index(drop=True)
user_114920240
Related