Python data source - first two columns disappear

Viewed 25

I have started using PowerBI and am using Python as a data source with the code below. The source data can be downloaded from here (it's about 700 megabytes). The data is originally from here (contained in IOT_2019_pxp.zip).

import pandas as pd
import numpy as np
import os

path = /path/to/file

to_chunk = pd.read_csv(os.path.join(path,'A.txt'), delimiter = '\t', header = [0,1], index_col = [0,1], 
                       iterator=True, chunksize=1000)

def chunker(to_chunk):
    to_concat = []
    for chunk in to_chunk:
        try:
            to_concat.append(chunk['BG'].loc['BG'])
        except:
            pass
    return to_concat
    
A = pd.concat(chunker(to_chunk))

I = np.identity(A.shape[0])

L = pd.DataFrame(np.linalg.inv(I-A), index=A.index, columns=A.columns)

The code simply:

  1. Loads the file A.txt, which is a symmetrical matrix. This matrix has every sector in every region for both rows and columns. In pandas, these form a MultiIndex. enter image description here
  2. Filters just the region that I need which is BG. Since it's a symmetrical matrix, both row and column are filtered.
  3. The inverse of the matrix is calculated giving us L, which I want to load into PowerBI. This matrix now just has a single regular Index for sector. enter image description here

This is all well and good however when I load into PowerBI, the first column (sector names for each row i.e. the DataFrame Index) disappears. When the query gets processed, it is as if it were never there. This is true for both dataframes A and L, so it's not an issue of data processing. The column of row names (the DataFrame index) is still there in Python, PowerBI just drops it for some reason.

what I see in PBI after I load the data with the code above

I need this column so that I can link these tables to other tables in my data model. Any ideas on how to keep it from disappearing at load time?

1 Answers

For what it's worth, calling reset_index() removed the index from the dataframes and they got loaded like regular columns. For whatever reason, PBI does not properly load pandas indices.

For a regular 1D index, I had to do S.reset_index().

For a MultiIndex, I had to do L.reset_index(inplace=True).

Related