How to handle MemoryError when using nested for-loop in Python

Viewed 22

I have a script like below

# Import phrases
search_phrases = pd.read_excel(r'P:\Phrases.xlsx')

# Functions
def string(df, category, group, field):
    phrase_list = df.loc[
        (
            (df['Category'] == category)
            & (df['Group'] == group)
        ), 'Value']

    phrase_string = F"' OR {field} LIKE '".join(phrase_list)

    return F"({field} LIKE '" + phrase_string + "')"

def extract(category, group):    
    logger.info(F'Extracting {category} clients with {group}')
    sql  = F"""
     SELECT 
           ID,
           '{group}'                       AS 'Group',
           '{category}'                    AS Category,
           Referral     
    FROM   
          TableT t 
          INNER JOIN TableC c
          ON t.ID = c.ID
    WHERE  
          (
           {string(search_phrases, category, group, 'c.Name')}
             )   
    """
    return cnxn.select(sql)

# Extract data
categories = search_phrases['Category'].drop_duplicates().copy()
groups = search_phrases['Group'].drop_duplicates().copy()

client_list = []
for category in categories:
    for group in groups:
        client_list.appendextract(category, group)
        
all_clients = pd.concat(client_list)

search_phrases is a table imported from Excel

Category            Group Value
0     Good              Low    A%
1     Good              Low    B%
2     Good           Normal    C%
3     Good           Normal    D%
4      Bad  Borderline High    E%
5      Bad  Borderline High    F%
6      Bad             High    G%

Data is extracted from SQL Database.

My problem is the above script will extract data like the folowing:
*Extracting Good clients with Low
*Extracting Good clients with Normal
*Extracting Good clients with Borderline High
*Extracting Good clients with High...

My actual 'search_phrases' table includes 200 rows. Hence I ran into MemoryError when running the script using nested for loops.

Is there any effective way to do that?
E.g.
*Extracting Good clients with Low
*Extracting Good clients with Normal
*Extracting Bad clients with Borderline High
*Extracting Bad clients with High

Any help would be greatly appreciated!

0 Answers
Related