Tokenizasyon

Tokenization is the process of converting raw text to the numerical token array, which a language model can work, and is the most basic pretreatment step of large language model architecture. Simple approaches can divide text directly into words or characters, but these methods have severe drawbacks: word-based tokenization requires a huge and insufficient dictionary, while character-based tokenization makes it difficult to extend the array to unnecessary location and use the context of the model efficiently. Therefore, today, the sub-word (subword)-based algorithms such as Byte-Pair Encoding (BPE), WordPiece and SentencePiece are preferred; these are often combined with the doubles of the passing character or character sequence, often forms a balanced dictionary that divides words into multiple meaningful parts.

A well-designed tokenization allows the model to represent both common and rare words, even words that they never see before (e.g. special names or technical terms) with a reasonable number of tokens; which directly affects training efficiency and model performance. The quality of tokenization varies from language to language at the same time: most popular tokenizers are optimized with English-weight data, text in English-like inserting languages are usually divided into more tokens, which increases the cost and narrows the active context window.