Karma Uzmanlar Modeli
The efficient model architecture, which is activated only by the corresponding 'uzman' sub-networks.
Mixed Experts (Mixture of Experts, MoE) architecture is an efficient nervous network design that consists of a large number of "graman" sub-networks of a model, not only a few of these experts for each input. In the model there is a "administrative" (router); this component decides which experts are most suitable for each token from, and only executes calculation through selected experts. Thus, even if the model has a total of billions of parameters, only a small part of these parameters is actively used during an outlook (inference).
The importance of this approach comes from the fact that the calculation cost is linearly increased by the number of parameters in the "intens" models, while increasing the capacity of MoE, which can greatly keep the cost of the interest. Google’s Switch Transformer and Mistral’s Mixtral model are examples that are popular with MoE architecture; some top-end models are thought to use this architecture. MoE is able to train and operate large-scale models more economically, while experts bring engineering challenges such as load imbalance and training instability.
The importance of this approach comes from the fact that the calculation cost is linearly increased by the number of parameters in the "intens" models, while increasing the capacity of MoE, which can greatly keep the cost of the interest. Google’s Switch Transformer and Mistral’s Mixtral model are examples that are popular with MoE architecture; some top-end models are thought to use this architecture. MoE is able to train and operate large-scale models more economically, while experts bring engineering challenges such as load imbalance and training instability.