Crop nan rows and columns of a matrix, but keep it square

I have a square matrix with > 1,000 rows & columns. In many fields at the "border" there is nan, for example:

grid = [[nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan],
        [nan, nan,   1, nan, nan],
        [nan,   2,   3,   2, nan],
        [  1,   2,   2,   1, nan]]

Now I want to eliminate all rows and columns where I only have nan. This would be the 1. and 2. row and the last column. But I also want to receive a square matrix, so the number of the eliminated rows must be equal to the number of eliminated columns. In this example, I want to get this:

grid = [[nan, nan, nan, nan],
        [nan, nan,   1, nan],
        [nan,   2,   3,   2],
        [  1,   2,   2,   1]]

I'm sure I could solve this with a loop: check every column & row if there is only nan inside and in the end I use numpy.delete to delete the rows & columns I found (but only the minimal number, because of getting a square). But I hope anyone can help me with a better solution or a good library.