Optimize log entropy calculation in sparse matrix

I have a 3007 x 1644 dimensional matrix of terms and documents. I am trying to assign weights to frequency of terms in each document so I'm using this log entropy formula http://en.wikipedia.org/wiki/Latent_semantic_indexing#Term_Document_Matrix (See entropy formula in the last row).

I'm successfully doing this but my code is running for >7 minutes. Here's the code:

int N = mat.cols();
for(int i=1;i<=mat.rows();i++){
    double gfi = sum(mat(i,colon()))(1,1); //sum of occurrence of terms
    double g =0;
    if(gfi != 0){// to avoid divide by zero error

        for(int j = 1;j<=N;j++){
            double tfij = mat(i,j);
            double pij = gfi==0?0.0:tfij/gfi;
            pij = pij + 1; //avoid log0
            double G = (pij * log(pij))/log(N);
            g = g + G;
        }
    }

    double gi = 1 - g;
    for(int j=1;j<=N;j++){
        double tfij = mat(i,j) + 1;//avoid log0
        double aij = gi * log(tfij);
        mat(i,j) = aij;
    }
}

Anyone have ideas how I can optimize this to make it faster? Oh and mat is a RealSparseMatrix from amlpp matrix library.

UPDATE Code runs on Linux mint with 4gb RAM and AMD Athlon II dual core

Running time before change: > 7mins

After @Kereks answer: 4.1sec