What methods do you use to handle out-of-vocabulary words or tokens during text generation in GPT models

Question

Can you suggest me 3 efficient techniques name to handle out of vocabulary words or tokens during text generation in GPT models?

Answer 1

The three efficient techniques are as follows:

• 1.Subword Tokenization(Byte pair encoding)
• 2.Character-Level Tokenization
• 3.Use of a special Token

These methods helps in mitigating issues related to out-of-vocabulary words improving functioning of the model during inference.