I have a script like below
# Import phrases
search_phrases = pd.read_excel(r'P:\Phrases.xlsx')
# Functions
def string(df, category, group, field):
phrase_list = df.loc[
(
(df['Category'] == category)
& (df['Group'] == group)
), 'Value']
phrase_string = F"' OR {field} LIKE '".join(phrase_list)
return F"({field} LIKE '" + phrase_string + "')"
def extract(category, group):
logger.info(F'Extracting {category} clients with {group}')
sql = F"""
SELECT
ID,
'{group}' AS 'Group',
'{category}' AS Category,
Referral
FROM
TableT t
INNER JOIN TableC c
ON t.ID = c.ID
WHERE
(
{string(search_phrases, category, group, 'c.Name')}
)
"""
return cnxn.select(sql)
# Extract data
categories = search_phrases['Category'].drop_duplicates().copy()
groups = search_phrases['Group'].drop_duplicates().copy()
client_list = []
for category in categories:
for group in groups:
client_list.appendextract(category, group)
all_clients = pd.concat(client_list)
search_phrases is a table imported from Excel
Category Group Value
0 Good Low A%
1 Good Low B%
2 Good Normal C%
3 Good Normal D%
4 Bad Borderline High E%
5 Bad Borderline High F%
6 Bad High G%
Data is extracted from SQL Database.
My problem is the above script will extract data like the folowing:
*Extracting Good clients with Low
*Extracting Good clients with Normal
*Extracting Good clients with Borderline High
*Extracting Good clients with High...
My actual 'search_phrases' table includes 200 rows. Hence I ran into MemoryError when running the script using nested for loops.
Is there any effective way to do that?
E.g.
*Extracting Good clients with Low
*Extracting Good clients with Normal
*Extracting Bad clients with Borderline High
*Extracting Bad clients with High
Any help would be greatly appreciated!