Bigrams Modelling: Text Length

davide cortellino 0
Viewed 32

  • I find myself in the process of extracting bigrams (sequences of two characters) from text data to perform an analysis of frequencies and develop a model

  • Now, the frequencies and related quintiles are of course affected by the text length, which can vary both in the train set and in the future score sets

  • I searched some proper approach to follow to avoid the variance in text length to affect the model performances.

It seems that one approach is the following:

“ Recall that for batching, we need to have all the sequences in a given batch be of uniform length ( N of words).

To do that, we either:

• (1) pad the sequences that are shorter than a given length or

• (2) truncate the sequences that are bigger than the given length.

• The question is how do we decide this length? We have several options:

We decide on a global maximum sequence length based on the sequence length characteristics of the training data. “

Meaning that:

  1. It is identified in the training set, within the distribution of N of words composing a text, the maximum value or 75% quantile , and use it as upper limit
  2. Based on this value, all the texts with greater length are truncated

The approach seems sound to me in avoiding that text length will affect the bigrams distributions and model performance on the score set

Still, I would know if someone has managed similar tasks before and if any relevant material can be suggested

text-mining nlp
1 Answers

One approach could be to split your dataset by chunks of text lengths, & compute tf-idf (well, or just ngrams) on each sizes, and do that in a multi-level, hierarchical, with levels of separations, like one level of chunking by 5 words, one by 10 words, one by 15, and then, as input to your model, do a weighted average of the scores on the clusters of different levels.

The reason why we would do this is that the smaller the length clusters, the higher the variance, because the fewer data points you have. So either taking a middle cluster size, or doing a hand tuned or learned average of the scores over different precisions of length clusters might help with that.

Please investigate just using TF-IDF instead of ngrams, and not using length stuff, first.

Jules G.M.0
Related