Use Quantization on HuggingFace Transformers models

user_16852041 0
Viewed 134

I'm learning Quantization, and am experimenting with Section 1 of this notebook.

I want to use this code on my own models.

Hypothetically, I only need to assign to model variable in Section 1.2

# load model
model = BertForSequenceClassification.from_pretrained(configs.output_dir)
model.to(configs.device)

My models are from a different library: from transformers import pipeline. So .to() throws an AttributeError.

My Model:

pip install transformers

from transformers import pipeline

unmasker = pipeline('fill-mask', model='bert-base-uncased')
model = unmasker("Hello I'm a [MASK] model.")

Output:

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

How might I run the linked Quantization code on my example model?

Please let me know if there's anything else I should clarify in this post.

quantization bert-language-model huggingface-transformers deep-learning python
1 Answers

The pipeline approach won't work for Quantisation as we need the models to be returned. You can however, use pipeline for testing the original models for timing etc.

Quantisation Code:

token_logits contains the tensors of the quantised model.

You could place a for-loop around this code, and replace model_name with string from a list.

model_name = bert-base-uncased
tokenizer = AutoTokenizer.from_pretrained(model_name )
model = AutoModelForMaskedLM.from_pretrained(model_name)
    
sequence = "Distilled models are smaller than the models they mimic. Using them instead of the large " \
f"versions would help {tokenizer.mask_token} our carbon footprint."

inputs = tokenizer(sequence, return_tensors="pt")
mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
    
token_logits = model(**inputs).logits

# <- can stop here

Source

user_168520410
Related