How to understand byte pair encoding?

user_7294253 0
Viewed 1089

I read a lot of tutorial about BPE but I am still confuse how it works.

for example. In a tutorial online, they said the folowing :

Algorithm

Prepare a large enough training data (i.e. corpus)

Define a desired subword vocabulary size

Split word to sequence of characters and appending suffix “” to end of

word with word frequency. So the basic unit is character in this stage. For example, the frequency of “low” is 5, then we rephrase it to “l o w ”: 5 Generating a new subword according to the high frequency occurrence. Repeating step 4 until reaching subword vocabulary size which is defined in step 2 or the next highest frequency pair is 1.

Taking “low: 5”, “lower: 2”, “newest: 6” and “widest: 3” as an example, the highest frequency subword pair is e and s. It is because we get 6 count from newest and 3 count from widest. Then new subword (es) is formed and it will become a candidate in next iteration.

In the second iteration, the next high frequency subword pair is es (generated from previous iteration )and t. It is because we get 6count from newest and 3 count from widest.

I do not understand why low is 5 and lower is 2:

does this meand l , o, w , lo, ow + = 6 and then lower equal two but why is not e, r, er which gives three ?

scikit-learn vectorization nlp python
1 Answers

The numbers you are asking about are the frequencies of the words in the corpus. The word "low" was seen in the corpus 5 times and the word "lower" 2 times (they just assume this for the example).

In the first iteration we see that the character pair "es" is the most frequent one because it appears 6 times in the 6 occurrences of "newest" and 3 times in the 3 occurrences of the word "widest".

In the second iteration we have "es" as a unit in our vocabulary the same way we have single characters. Then we see that "est" is the most common character combination ("newest" and "widest").

user_43748620
Related