Text augmentation effective when using Doc2Vec

Viewed 50

There are many ways to augment text data, and many articles about this area. I would like to understand if there is a text augmentation technique which is likely to work well, when the text input is vectorized using a Doc2Vec model.

My intuition here would be that using synonym replacement may work well, to create a smoother decision boundary in downstream classifiers.

The pipeline is as follows: text -> d2v -> clf

EDIT: based on comments. I am not looking to augment prior to d2v training. I am using my trained model for a downstream task which looks like this:

text -> d2v -> binary classifier

Lets say I am predicting sentiment, but I have only a few positive samples:

"the dog is happy"

So I am making more samples by augmenting the positive:

"the dog is ecstatic"

Now I have two samples which I pass to d2v to vectorize and use downstream rather than one.

My question is therefore what sort of augmentation works, because for example since d2v in some modes does not care much about word order, doing things like "sentence swap" could be useless.

1 Answers

Results are always sensitive enough to the quality of your data, choice of metaparameters, & exact project goals that the only way to know the answers to such questions is to try it.

I suspect that in general, many domain-sensitive augmentation techniques could help Doc2Vec training.

As one example, if you know that in your problem domain, certain multi-word phrases are extra-significant, either combining the unigrams, or adding a synthetic token for the phrase.

Or similarly, the synonym-replacement you mention might be helpful, especially in cases where (a) the words are strong synonyms; and (b) each alternate token appears in only a few, or individually-idiosyncratic, contexts. Then, there's a risk that the individual variants don't survive the min_count cutoff, or each's idiosyncratic usage dominates the resulting model's learning. Canonicalizing the tokens into one form then injects extra, valuable domain knowledge – these words should be the same, even if they look different.

But what if instead there are relevant shades-of-different meaning to the alternate tokens? And further, what if the training data is plentiful – so that there are plenty of usage contexts with which to detect both the tokens' commonalities, and their contrasts?

In that case, the premature synonym-replacement could hurt. It'd discard usable info in the original training data. The essence of Doc2Vec (like other word2vec-related algorithms) is that by forcing a smaller model to describe a larger training set, intrinsic generalizable relationships – like close or partial synonymy – can be found by the model, given enough data & suitable model constrain parameters.

So you should try replacement-or-not, and even partial-replacement at various-thresholds, then check the end results against a robust repeatable evaluation, to know which tactics help and which hurt.

In particular, I might try some of the Doc2Vec modes that co-train individual word-vectors – either plain default dm=1 or adding dbow_words=1 to dm=0. Then, at the end of a training test, check whether the words you were considering to coalesce into a sngle token already have very-similar word-vectors. If they're already close, the preprocessing replacement might be superfluous. If they're far, then maybe (a) replacement might help, if the words should be truly alike in meaning; but also (b) your assumption that the words should have identical meaning might be wrong, and the data is warning you that in real usage, the words have different significance.

In either case, obtaining more data, if possible, might 'iron-out' any false contrasts, or reinforce significant contrasts, better than improvisational manual-augmentation. And proper testing of different Doc2Vec models in the downstream classifier should be the ultimate test of what pre-Doc2Vec text prep is best.

Related