Pick equal number of examples from each category that fits within the context length

user_2195440 0
Viewed 220

Pick all lines with the smallest word_count from a CSV file in a round-robin fashion based on type (so that all types are present as much as possible) that fit within the context length.

def get_line_from_input_csv_with_in_context_length(input_file, context_length):

  • context_length = 2001

I have a CSV file containing the following fields:

line_no,type,word_count
1,assertEquals,704
2,assertThat,303
3,assertNotNull,159
4,assertThat,348
5,assertFalse,186
6,assertArrayEquals,699
7,assertEquals,119
8,assertTrue,314
9,assertEquals,552
10,assertSame,151
11,assertEquals,110
12,assertEquals,210
13,assertTrue,807
14,assertTrue,1231
15,assertEquals,541
16,assertEquals,1435
17,assertEquals,305
18,assertEquals,134
19,assertArrayEquals,641
20,assertEquals,800
21,assertEquals,522
22,assertEquals,150
23,assertThat,608
24,assertArrayEquals,280
25,assertEquals,245
26,assertEquals,364
27,assertEquals,637
28,assertEquals,615
29,assertThat,362
30,assertTrue,1468

Here expected output:

assertEquals, line_no= 110, 119, 134, 245
assertThat, line_no= 303
assertNotNull, line_no= 159
assertFalse,  line_no= 186
assertArrayEquals, line_no= 280
assertSame, line_no= 151
assertTrue, line_no=314

I know how to do it in a brute force way i.e.:

  • Grouping the input file based on type and putting it into a list of a multi-dictionary.
  • Then, sort each entry of that multi-dictonary i.e. for each type based on word_count and put that in a list.
  • Now iterate list of list and pick the smallest until I hit the context_length.

But I am sure there is a more pythonic way to achieve this. Can anyone suggest me any pointers?

pandas dataframe python-3.x group-by python
2 Answers

Using pandas, we can group everything by type and then rank all the elements in each group according to word count. Then we can sort all these by rank and word_count to get them into the proper order. Once they are sorted, we do a cumsum() to get the running total to compare against context_length. And finally do another group by to get it listed out by type:

import pandas as pd

df = pd.read_csv(csv_io, index_col=0)
context_length = 2001

df['rank'] = df.groupby("type")['word_count'].rank(method="first", ascending=True)
df = df.sort_values(['rank', 'word_count'])
df['word_count_cumsum'] = df['word_count'].cumsum()

df = df[df['word_count_cumsum'] < context_length]

types_list = df.groupby('type')['word_count'].apply(list)

print(types_list)

Output looks like:

type
assertArrayEquals         [280]
assertEquals         [110, 119]
assertFalse               [186]
assertNotNull             [159]
assertSame                [151]
assertThat           [303, 348]
assertTrue                [314]
Name: word_count, dtype: object
user_64104500

As example, you can use this:

  1. Read data from CSV file and sort it by word_count,
    and next group by type.
  2. Pick the first element based on var_count, while you can pick it: 
    var_count = [[i,0,csv_data[i]] for i in type]
var_count = sorted(var_count,key = lambda i:i[2][i[1]])
while cnt < context_length:
    i = var_count[0]
    cur_group = i[0]#Current Group
    cur_len = i[2][i[1]]#Current Length
    var_count[0][1] += 1
    #Paste your code ONLY here
    var_count = sorted(var_count,key = lambda i:i[2][i[1]])
user_195475220
Related