Dataframes with regex search to build matching items dataframe

Viewed 49
  1. I pull all the tables and columns into a dataframe:
cf = pd.read_sql_query('SELECT DISTINCT TABLE_NAME,COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = \'dbo\'', engine)
  1. I have a table of queries I extracted from Crystal Reports and pulled into a second dataframe which returns two columns SQL, FilenameCrystalReport

I need to iterate over each of the SQL queries stored using a simple re.search(tablename, sql) to build a dictionary of all the tables used in the query. Then I want to get all the column names do a similar search re.search(columnname, sql). For all the matches of those iterations, I want to build a final result dataframe or return a dictionary.

desired output

FilenameCrystalReport_A, TableNameA, Column A
FilenameCrystalReport_A, TableNameA, Column B
FilenameCrystalReport_A, TableNameA, Column C
FilenameCrystalReport_A, TableNameA, Column D
FilenameCrystalReport_A, TableNameA, Column E
FilenameCrystalReport_B, TableNameA, Column A
FilenameCrystalReport_B, TableNameA, Column B
FilenameCrystalReport_B, TableNameZ, Column A
FilenameCrystalReport_B, TableNameD, Column A

I am currently at the following:

#start looping through the queries

for i in range(len(qf)):
    q = qf.iloc[i,0]

    #looping through all tables in database
    for j in range(len(df)):
        t = df.iloc[j,0]
        #print(t)
        match = re.search(t,q)

        if match is not None:
            #print(match.group())
            #break

            #clean report names imported from query

            s = qf.iloc[i,1]
            if s:
                s1 = s.strip()
                s1 = s1.replace("\\N","N")
                s1 = s1.replace("\\C","C")
                print(s1,'TABLES',match.group())

    for k in range(len(cf)):

        c = cf.iloc[k,0]
        #print(t)
        match = re.search(c,q)

        if match is not None:
            #print(match.group())
            #break

            #clean report names imported from query

            s = qf.iloc[i,1]
            if s:
                s1 = s.strip()
                s1 = s1.replace("\\N","N")
                s1 = s1.replace("\\C","C")
                print(s1,'COLUMNS',match.group())

Where I am stuck at is the having both table_name and column in the query and pull into dataframe. I cannot get a distinct table_name and thus multiple returns due to multiple columns associated with the table.

When I do two queries, I lose the continuity with the column name to a table in question. I need those two to be tied together. So when I pull everything back I have the report name, the table names, and the columns associated with those tables.

0 Answers
Related