I'm trying to implement a HuggingFace model in a Flask API. I've seen several posts explaining premade solutions for generating predictions, but that doesn’t fit my needs; I need to call native HuggingFace methods, as in below:
def rate_offensiveness(sentence):
"""sentence through offensiveness model and returns probability from 0-1"""
encoded_input = tokenizer(sentence, return_tensors='pt')
output = model(**encoded_input)
results = softmax(output[0][0].detach().numpy())
sentence_labels_dict = {}
ranking = np.argsort(results)
ranking = ranking[::-1]
for i in range(results.shape[0]):
label = labels[ranking[i]]
sentence = results[ranking[i]]
sentence_labels_dict[label] = np.round(float(sentence), 4)
return sentence_labels_dict['offensive']
I've been trying to run it in Heroku with the model in the slug (in the Github repo), but I run into the problem of the slug size being too big. I've also tried storing it in S3 and loading it into local ram on Heroku, and I run into the same problem.