Most efficient way to apply a function and query for values on a table of million/billion rows

I have parquet files containing million/billion rows, I'm trying to find a faster process of applying a function and querying for values on these big tables. Example ipynb code of a 1M-rows table:

from pyarrow import parquet as pq
from ttictoc import TicToc #Preferred time tracker
t = TicToc() 

#Function to be applied
def val_calc(x,n_cols): 
    import random
    import string

    #Mapping process
    abc_vals = string.ascii_uppercase[:n_cols] #N-alphabets in the df
    x_map = {i:random.randint(j,n_cols+10) for i,j in zip(abc_vals,range(n_cols))} #map-dict

    #Calculations... Formula: Σ(Xi*Yi)/ΣY 
    y_vals = {i:j**2 for i,j in x_map.items()} #Y's set of values to use 

    weights = [x_val*y_val for x_val,y_val in zip(x_map.values(),y_vals.values())] #(Xi*Yi)

    result = sum(weights)/sum(y_vals.values()) #Σ(X*Y)/ΣY 

    return result

#Getting the parquet file
file_path = 'C:/XYZ/project/'
file_name = 'gx6c'

#pyarrow parquet -> pandas
large_pq = pq.read_table(file_path+file_name+'.pq').to_pandas()

#Number of columns - column per alphabet in a row: 
n_columns = int(file_name.split('x')[-1].replace('c',''))

#-------------------------------------------------------Results and time taken
t.tic() #start time
                              #Function applied
large_pq['values'] = large_pq.apply(lambda x: val_calc(x, n_columns), axis=1) 

t.toc() #end time

print(f'Time passed for applying function: {round(t.elapsed,5)} seconds')
display(large_pq)


#Querying part
t.tic()
queried = large_pq[large_pq['values'].between(12,13)]
t.toc()

print(f'Time passed for query: {round(t.elapsed,5)} seconds')
display(queried)

The output:

Time passed for applying function: 17.60126 seconds
    
    abc     values
0   AAAAAA  13.258228
1   AAAAAB  10.227642
2   AAAABA  11.264317
3   AAABAA  12.422303
4   AABAAA  13.537634
... ... ...
999995  JJIJJJ  12.620214
999996  JJJIJJ  11.323636
999997  JJJJIJ  10.756757
999998  JJJJJI  10.358811
999999  JJJJJJ  10.896328
1000000 rows × 2 columns

Time passed for query: 0.04801 seconds
    
    abc     values
3   AAABAA  12.422303
5   ABAAAA  12.062818
13  AAAAAD  12.762040
16  AADAAA  12.925373
25  AAAAAF  12.661267
... ... ...
999967  IJJJII  12.936667
999972  JIJIJI  12.331742
999986  JIJJJI  12.133333
999993  IJJJJJ  12.179487
999995  JJIJJJ  12.620214
284129 rows × 2 columns

Repeating the same for a 13M-rows table with 17 alphabets per row (or 17 "columns") took around 10 minutes, the query step 0.22524 seconds. With larger files I get memory errors, so a billion-rows mark can't be reached. Would there possibly be any work-arounds to perform the processes in a much smaller time frame, like 10 seconds instead of minutes for a 13M-rows table?