Converting pseudo algorithm to python - pandas code

Question

I am trying to convert a pseudo code to pandas code. Would appreciate any help or guidance.

General idea is to come up with a function f to select rows from a toy example dataset which -> has 100 rows and 5 columns ["X", "Y", "Z", "F", "V"] randomly filled with numbers between [0, 500]. Apart from the data, the second input for the function is the columns cols_to_use it should use in selection for which default is to use all of them.

Description. Goal is to select 10 rows from the sample dataset. There are 5 probabilities for the second argument of the function -> selection based on [1, 2, 3, 4, 5] columns.

If all columns have to be used, then we select 2 rows per column. We select the rows corresponding to top 2 values per each column. There can be rows which will overlap during the initial selection. Lets call it overlap1 event. If overlap1 event happens, we randomly select a column for which we keep the overlapped row(s) and for the other(s) we add the 3rd. During this process there can be overlaps as well with the new selected ones and already selected ones -> call it overlap2 event. If overlap2 happens, use top 4th, top 5th and etc rows for that column. There is on average .25 probability that there will be at least one overlap during initial selection, so this is quite important to account for. Final selection must consist of 10 unique rows.

if there are 4 columns to base the selection upon, we select rows corresponding to the top 2 values per each of the columns and solve overlap1 event. But we still need to select 2 more rows. So, we randomly draw 2 columns from those 4 and for them we select additional row -> corresponding to the 3rd, or when overlap2 happens to the 4th and etc.

if there are 3 columns, select 3 rows per column as per the rule aforementioned + overlap1 solution if any, and randomly select a column for which we should add the remaining 1 option + solve overlap2 event

when 2 columns must be used, select 5 rows per column + overlap 1 and 2 events

when only 1 column must be used select top 10 rows corresponding to highest 10 values for that column

# sample dataset to work with

sample = pd.DataFrame(np.random.randint(0, 500, size = (100, 5)))
sample.columns = "X Y Z F V".split()

# the function I have written so far
def f(df, cols_to_use = ["X", "Y", "Z", "F", "V"]):

    how_many_per_feature = {
        5:2,
        4:2,
        3:3,
        2:5,
        1:10
    }
    n_per_group = how_many_per_feature[len(cols_to_use)]

    # columns to randomly choose when adding extra options
    # could not find a proper way to implement this
    
    if len(cols_to_use) == 4:
        randomly_selected_columns = random.sample(cols_to_use, 2)
    elif len(cols_to_use) == 3:
        randomly_selected_columns = random.sample(cols_to_use, 1)
    
    
    # first I filter the dataframe on columns I need
    filtered_df = df[cols_to_use]
    
    # using pandas melt to select top n_per_group
    
    
    result = col_filtered.melt(id_vars = "obj_function",
                        var_name = "feature",
                        ignore_index = False,
                        ).groupby("feature").value.nlargest(n_per_group)

    # here supposed to handle overlap1 events
    
    # here overlap2 events
                        
    index = result.reset_index().level_1.values
    
    return df.iloc[index,:]

I could not implement the dynamic selection based on overlap events' handling.

user_4133131 · Answer

Something like this, perhaps? It's not the cleanest, but I think it gets the work done.
I was not clear on your intentions by There are 5 probabilities for the second argument of the function -> selection based on [1, 2, 3, 4, 5] columns. though.
The idea here is to separate the columns and sort each individually. We can then select and remove indices as needed for each column. Checking for overlaps is done by checking common indices between any possible combination of columns, where we keep iterating until all overlaps are resolved (i.e. solve for overlap3, overlap4 etc events).

from itertools import combinations
from functools import reduce
import numpy as np

def f(df, cols_to_use=None, n_rows=10):
    df = df.copy()  # Ensure we don't modify the original dataframe
    cols_to_use = cols_to_use or ["X", "Y", "Z", "F", "V"]
    n_rows_per_col = n_rows // len(cols_to_use)
    additional_rows = n_rows % len(cols_to_use)
    top = dict()
    indices = dict()

    # Sort and create initial indices set
    for col in cols_to_use:
        top[col] = df[col].sort_values(ascending=False)
        indices[col] = top[col].iloc[:n_rows_per_col].index
        top[col] = top[col].drop(index=indices[col])  # Remove added indices

    # Ensure we have exactly n_rows indices
    if additional_rows:
        # select additional_rows columns from our cols_to_use
        more_cols = np.random.choice(cols_to_use, size=additional_rows)
        for c in more_cols:
            indices[col] = indices[col].union(top[col].iloc[[0]].index)
            top[col] = top[col].drop(index=indices[col], errors='ignore')

    # Resolve overlap events as needed
    combs = list(combinations(cols_to_use, r=2))
    overlap = True
    while overlap:
        overlap = False
        for col1, col2 in combs:
            intersect = indices[col1].intersection(indices[col2])
            if intersect.shape[0]:  # Overlap between col1 and col2
                overlap = True
                # Consider all columns that contain this intersect
                cols = [c for c in cols_to_use if intersect.isin(indices[c]).any()]
                # Choose which column to remove the overlap from and use the following entries from its top value
                c = np.random.choice(cols)
                indices[c] = indices[c].drop(intersect).union(top[c].iloc[:intersect.shape[0]].index)
                top[c] = top[col].drop(index=indices[c], errors='ignore')
    return df[cols_to_use].loc[reduce(lambda x, y: x.union(y), indices.values())]

EDIT: Ensured all relevant columns in every overlap event are available for random selection, regardless if they're part of the combs that identified the overlap event.

user_15529253 · Answer

The idea is to have the row order according to the columns in the variableranking_list . Then compare the row selection for each column with the rows of the other columns and if ever there is an overlap then I randomly delete the row in the ranking of one of the two columns

import pandas as pd
import numpy as np 
import random
  
def f(df, cols_to_use = ["X", "Y", "Z", "F","V"]):
    
    
    n_col = len(cols_to_use)
    n_row_per_col = 10 // n_col
    n_row_add = 10 % n_col
    lim_col = [n_row_per_col] * n_col
    for i in random.sample(range(n_col), n_row_add): 
        lim_col[i]+=1
    
    
    
    ranking_list = []
    
    for i in cols_to_use : 
        ranking_list.append(list(df.sort_values(by=i, ascending=False).index))


    key = True

    while(key == True ) :

        key = False 

        for i in range(n_col) : 

            for l in range(n_col): 

                if (i != l) : 

                    overlap = set(ranking_list[i][:lim_col[i]]).intersection(set(ranking_list[l][:lim_col[l]]))

                    if ( len(overlap) > 0 ) : 
                        key = True
                        for r in list(overlap): 
                            random_col = random.choice([i,l])
                            ranking_list[random_col].remove(r)
                            
                            
    
    list_of_index = []
    for i in range(n_col) : 
            list_of_index.append(ranking_list[i][:lim_col[i]])

    list_of_index = [item for sublist in list_of_index for item in sublist]

    return(df.iloc[list_of_index])

Converting pseudo algorithm to python -> pandas code

2 Answers