Token
The smallest meaning that a language model uses when processing the text.
Token is the smallest meaning or trading unit that a language model uses when processing text. A token does not always correspond to a word; while short and frequently used words can be a single token, long or rare words can be divided into multiple sub word parts (subword); in some cases a single character or pointing sign can also be a token. For example, when "pay" can be a single token, a less common word can be divided into two parts such as "yap" and "#ay". This sub-word approach allows the model to process new or rare words that are not included in the dictionary.
The text of large language models does not directly process; the text with a process called tokenization, then converts each token to a numerical entity. The model works by predicting the next token through this token sequence. The concept of token is critical in practical aspect: most commercial artificial intelligence is made according to the number of tokens processed by the pricing of the API, and the maximum text length that a model can work at one time is measured in tokens. As a general rule, about four characters in English are equivalent to a token, the number of tokens can be higher than the word, in addition to English.
The text of large language models does not directly process; the text with a process called tokenization, then converts each token to a numerical entity. The model works by predicting the next token through this token sequence. The concept of token is critical in practical aspect: most commercial artificial intelligence is made according to the number of tokens processed by the pricing of the API, and the maximum text length that a model can work at one time is measured in tokens. As a general rule, about four characters in English are equivalent to a token, the number of tokens can be higher than the word, in addition to English.
Interactive experiment preparing...