Convert multiple unnamed lists as a result of batch size predictions to Pandas Dataframe

user_14015093 0
Viewed 50
merchant_labels = [165 different labels in total]

batch_size = 10 # see how big you can make this number before OOM
sequences = df1['clean_text'].to_list()
zero_shot_labels = []
for i in range(0, len(sequences), batch_size):
    try:
        zero_shot_labels += classifier(sequences[i:i+batch_size], merchant_labels, multi_label=True) # every new batch sized prediction is under the same list name = zero_shot_labels
    except ValueError:
        pass

Concatenate all the lists generated from the batch predictions I cannot make the concatenation since every new list is under the name: zero_shot_labels. So when I try to unnest the lists to make a Pandas Dataframe, I only receive the results of my first 10 predictions. Every new list of dictionaries has the same name = zero_shot_labels. Is there an optimal way we make this concatenation, or another type of code where I can do batch predicts and get the final Dataframe with all the rows of all different predictions? Thanks in advance!

huggingface-transformers pandas python
1 Answers

use within the loop

zero_shot_labels.append(..)

to append the different dataframes to the list.

And concatenate the dataframes from the list with:

all_zero_shot_labels = pd.concat(zero_shot_labels, axis=0, ignore_index=True)
user_169573670
Related