It depends.
If SquareMatrix just adds one extra method to check whether it is diagonal, and all other methods can remain exactly as they where, then it might be OK. Adding methods in a subclass to an inherited class is generally compatible with LSP.
But most likely, you would also want to insist that the matrix is, in fact, square. So if your original Matrix class has functionality to change the height or the width of the matrix, you would have to somehow 'disable' this functionality in the SquareMatrix subclass. And such disabling is incompatible with LSP.
If you wish to have a separate SquareMatrix class, then the normal route would be a MatrixInterface with an API that all matrices have in common, an AbstractMatrix providing a basic implementation of this interface, a ResizableMatrix for matrices of which the dimensions can change, and a SquareMatrix with height and width guaranteed to be equal -- where the latter two both inherit from the AbstractMatrix.
If this is overly complex for your use case, a cleaner solution might be to add two methods to your Matrix class: isSquare, which returns whether or not a matrix is square, and isDiagonal, which under the assumption (precondition) that the matrix is square returns whether the matrix also diagonal.