- I pull all the tables and columns into a dataframe:
cf = pd.read_sql_query('SELECT DISTINCT TABLE_NAME,COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = \'dbo\'', engine)
- I have a table of queries I extracted from Crystal Reports and pulled into a second dataframe which returns two columns SQL, FilenameCrystalReport
I need to iterate over each of the SQL queries stored using a simple re.search(tablename, sql) to build a dictionary of all the tables used in the query. Then I want to get all the column names do a similar search re.search(columnname, sql). For all the matches of those iterations, I want to build a final result dataframe or return a dictionary.
desired output
FilenameCrystalReport_A, TableNameA, Column A
FilenameCrystalReport_A, TableNameA, Column B
FilenameCrystalReport_A, TableNameA, Column C
FilenameCrystalReport_A, TableNameA, Column D
FilenameCrystalReport_A, TableNameA, Column E
FilenameCrystalReport_B, TableNameA, Column A
FilenameCrystalReport_B, TableNameA, Column B
FilenameCrystalReport_B, TableNameZ, Column A
FilenameCrystalReport_B, TableNameD, Column A
I am currently at the following:
#start looping through the queries
for i in range(len(qf)):
q = qf.iloc[i,0]
#looping through all tables in database
for j in range(len(df)):
t = df.iloc[j,0]
#print(t)
match = re.search(t,q)
if match is not None:
#print(match.group())
#break
#clean report names imported from query
s = qf.iloc[i,1]
if s:
s1 = s.strip()
s1 = s1.replace("\\N","N")
s1 = s1.replace("\\C","C")
print(s1,'TABLES',match.group())
for k in range(len(cf)):
c = cf.iloc[k,0]
#print(t)
match = re.search(c,q)
if match is not None:
#print(match.group())
#break
#clean report names imported from query
s = qf.iloc[i,1]
if s:
s1 = s.strip()
s1 = s1.replace("\\N","N")
s1 = s1.replace("\\C","C")
print(s1,'COLUMNS',match.group())
Where I am stuck at is the having both table_name and column in the query and pull into dataframe. I cannot get a distinct table_name and thus multiple returns due to multiple columns associated with the table.
When I do two queries, I lose the continuity with the column name to a table in question. I need those two to be tied together. So when I pull everything back I have the report name, the table names, and the columns associated with those tables.