getting the index of the last non-empty value in pandas

Ahmed Nabil 0
Viewed 54

Let's assume I have the following data frame:

          x         y
1    -1.808909  0.093380
2     1.733595 -0.380938
3    -1.385898  0.714071

And I want to insert a value in the column after "y". However, it's possible that I might insert more than one value.

So, I need to check if the cell after "y" is empty or not to avoid overwriting the cell.

so, the expected output might be like

          x         y      
1    -1.808909  0.093380   5
2     1.733595 -0.380938   6    7
3    -1.385898  0.714071   8

Compared to the input above, I need to check the cell first if it's empty or not.

I thought I might use: x = df.iloc[1,:].last_valid_index()

but that method returns "y" not the index of "y" which is 1.

later I'll use that index to inset "5":

x +=1
df.iloc[1,x] = 5

I want to use that approach of finding the last non-empty cell because of the 2nd row in the output. You see that I need to insert "6" then "7" If I ended up using always the same method like this one:

    df.iloc[1,2] = 6
    df.iloc[1,2] = 7

It'll overwrite the "6" when inserting "7"

One more thing, I can't look for the value using something like: (df['y'].iloc[2]).index because later I'll have two "y" columns so, that might leads to returns index number less than the required.

dataframe pandas python
2 Answers

It is easy to identify the position of the first zero in each row in a numpy array or a dataframe. Let's create a dataframe with zeros after a certain position:

df = pd.DataFrame(np.random.normal(size=(5, 10)))
df

   0  1  2  3  4  5  6  7  8  9
0  4  1  4  2  6  0  0  0  0  0
1  5  4  9  5  5  4  0  0  0  0
2  6  6  6  5  4  8  6  0  0  0
3  5  3  9  5  3  9  6  3  0  0
4  3  2  7  9  7  6  6  7  5  0

For instance, the code below will give you all positions in the dataframe where the value is 0

np.argwhere(df.values == 0)

array([[0, 5],
       [0, 6],
       [0, 7],
       [0, 8],
       [0, 9],
       [1, 6],
       [1, 7],
       [1, 8],
       [1, 9],
       [2, 7],
       [2, 8],
       [2, 9],
       [3, 8],
       [3, 9],
       [4, 9]], dtype=int64)

Or you can get the positions where the values are not zero:

np.argwhere(df.values != 0)

array([[0, 0],
       [0, 1],
       [0, 2],
       [0, 3],
       [0, 4],
       [1, 0],
       [1, 1],
       [1, 2],
       [1, 3],
       [1, 4],
       [1, 5],
       [2, 0],
       [2, 1],
       [2, 2],
       [2, 3],
       [2, 4],
       [2, 5],
       [2, 6],
       [3, 0],
       [3, 1],
       [3, 2],
       [3, 3],
       [3, 4],
       [3, 5],
       [3, 6],
       [3, 7],
       [4, 0],
       [4, 1],
       [4, 2],
       [4, 3],
       [4, 4],
       [4, 5],
       [4, 6],
       [4, 7],
       [4, 8]], dtype=int64)

I hope it helps.

vtec0

I suggest this, a less complicated solution

import random
nums = [0, 7, 78, 843, 34893, 0 , 2, 23, 4, 0]
random.shuffle(nums)
thg = [x for x in nums if x != 0]
print(thg[0])

what this does is shuffle the 'nums' list and filters out all the zeros. Then it prints the first non-zero value

userX07m10
Related